The AVIs are submitted to an AVI catalogue where they are run near the Gaia data and they are stored in the user’s allocated user-space. These AVIs, as well as the output results of the analysis they are used for, can be shared with other GAVIP users using the shared data dashboard. Once AVIs are started they appear in the active window where they can be accessed and the jobs they are created to do can be done.
Jupyter notebooks can be executed within GAVIP by any user. They are started in an isolated container built from the Python AVI template; this permits a notebook to import and use features within the AVI framework (e.g. the TAP+ connector). The notebooks are saved in the user-space, and can be downloaded using the user-space browser.
When starting your notebook, you will be prompted for an upper limit on the resources to be allocated to the notebook; this helps the platform to control load when analysing the Gaia data-products. Note that high resource limits may take longer to guarantee depending on overall platform load, which may cause delays starting the Jupyter notebook.
As the use of Jupyter/IPython notebooks is increasingly common for general analysis, an instance of the Jupyter notebook server is provided with AVI container images.
Your notebooks, and all data-products are stored in a persistent volume known as the “user-space”. Notebooks may also be downloaded/uploaded or shared to other users via the user-space browser. For more details, see section User-Space.
A catalogue of AVI views will be provided for the user when they wish to add a new view. It is expected that these views will be identified when an AVI is submitted to GAVIP through a naming convention (e.g.
dashboard_3d_star_plot()); this is yet to be finalized.
Provides a persistent storage system for each user. The user-space is currently implemented as an NFS volume (though this is subject to change). Alternative methods of user-space storage may be object bases (similar to Amazon S3). Docker can be extended to use alternative storage engines through its plugin system.
All AVIs started by each user will be configured to mount their unique user-space for reading and writing data products. A user-space browser is included in the portal web system that enables users to browse, upload, download, and delete the corresponding files.
To ensure that GAVIP resources are not over-allocated, AVI pipelines are not started immediately; rather they are managed by a scheduler. The scheduler monitors various metrics of the platform including GAVIP resources, job submission time, jobs submitted per user, and begins an AVI pipeline when appropriate. An AVI pipeline is executed in a separate container to the AVI interface, but they are structured identically; the most significant differences are resource allocations and the processes they are running. Ultimately, the solution to resource management within GAVIP is to allow AVIs to run in di erent modes, run complex processing tasks with dynamically allocated resources, and defer those tasks until possible for safe execution.
Operators can see all AVI builds in the Developer view, but are not permitted to build/remove the releases.