Portal

The AVIs are submitted to an AVI catalogue where they are run near the Gaia data and they are stored in the user’s allocated user-space. These AVIs, as well as the output results of the analysis they are used for, can be shared with other GAVIP users using the shared data dashboard. Once AVIs are started they appear in the active window where they can be accessed and the jobs they are created to do can be done.

Jupyter Notebooks

Jupyter notebooks can be executed within GAVIP by any user. They are started in an isolated container built from the Python AVI template; this permits a notebook to import and use features within the AVI framework (e.g. the TAP+ connector). The notebooks are saved in the user-space, and can be downloaded using the user-space browser.

When starting your notebook, you will be prompted for an upper limit on the resources to be allocated to the notebook; this helps the platform to control load when analysing the Gaia data-products. Note that high resource limits may take longer to guarantee depending on overall platform load, which may cause delays starting the Jupyter notebook.

_images/jupyter.png

As the use of Jupyter/IPython notebooks is increasingly common for general analysis, an instance of the Jupyter notebook server is provided with AVI container images.

_images/jupyter_nb.png

Your notebooks, and all data-products are stored in a persistent volume known as the “user-space”. Notebooks may also be downloaded/uploaded or shared to other users via the user-space browser. For more details, see section User-Space.

Avi Usage

_images/aviusage1.png _images/aviusage2.png _images/aviusage3.png

Avi Catalogue

A catalogue of AVI views will be provided for the user when they wish to add a new view. It is expected that these views will be identified when an AVI is submitted to GAVIP through a naming convention (e.g. dashboard_3d_star_plot()); this is yet to be finalized.

_images/catalog.png

User-Space

Provides a persistent storage system for each user. The user-space is currently implemented as an NFS volume (though this is subject to change). Alternative methods of user-space storage may be object bases (similar to Amazon S3). Docker can be extended to use alternative storage engines through its plugin system.

_images/userspace.png

All AVIs started by each user will be configured to mount their unique user-space for reading and writing data products. A user-space browser is included in the portal web system that enables users to browse, upload, download, and delete the corresponding files.

Shared Data

To minimize unnecessary processing of data, users can share data products. Other users can reuse these data products in their AVIs. Analysis can take a long time. Data products are valuable. Supports reproducibility and increases AVI value. Each user has a volume for their data (user-space). NFS volume. AVIs persist data using this volume. Browser in GAVIP. Shared data is mounted, not copied. Data mounted (read-only). Prevents unnecessary copying. Shared data products increase AVI value. Supports analysis reproducibility.

_images/shared.png

Resource Allocation

To ensure that GAVIP resources are not over-allocated, AVI pipelines are not started immediately; rather they are managed by a scheduler. The scheduler monitors various metrics of the platform including GAVIP resources, job submission time, jobs submitted per user, and begins an AVI pipeline when appropriate. An AVI pipeline is executed in a separate container to the AVI interface, but they are structured identically; the most significant differences are resource allocations and the processes they are running. Ultimately, the solution to resource management within GAVIP is to allow AVIs to run in di erent modes, run complex processing tasks with dynamically allocated resources, and defer those tasks until possible for safe execution.

Caveats

Operators can see all AVI builds in the Developer view, but are not permitted to build/remove the releases.