AVI Development

Introduction

Welcome to the Added Value Interface (AVI) development documentation. In this chapter, we detail

  1. How an AVI is designed
  2. How the AVI works overall
  3. How the AVI interface works
  4. How the AVI pipelines work
  5. How to start an AVI in “standalone” mode on your machine
  6. How to submit an AVI to GAVIP

Go to the Quickstart section to start up an AVI in 3 steps, and 1 command.

Development Requirements

  • A Docker compatible operating system
  • Docker
  • git

Quickstart

  1. Install Docker for your Operating System.

  2. Install the GAVIP Client (available on livelink)

  3. Run the quickstart command with your own parameters

    gavip_client development quickstart_avi

GAVIP Client quickstart example:

gavip_client development quickstart_avi --avi-dir="/tmp" --avi-name=quickstart --example-reponame=example-avi-simple --avi-template=ps_avi_python_2:0.3.0
–avi-dir:The directory to create your AVI and Data volume
–example-reponame:
 The example repository to take the AVI code from (see our examples on github)
–avi-template:The AVI base image or “template” to use. Certain templates provide additional features and runtimes such as Java 8 or Python 3.

AVI Components

An AVI is an application running in a Docker container. It is designed to provide a reusable and reconfigurable tool that other scientists can use, contributing to an expanding catalogue of tools which operate on the Gaia archive. It is expected that analysis will first be developed using Jupyter notebooks, and then evolved to create an AVI.

The Docker container (known as AVI container, shown in purple below) includes a number of libraries and programming runtimes. The AVI code is executed within the AVI framework (shown in green) which handles operations such as authentication and pipeline management. The AVI code is the interface and pipeline provided by the developer; e.g. transient analysis pipelines, or data visualization tools. Each of these elements are elaborated in the following sections.

The diagram below provides a high level overview of an AVI. It is important to note that the AVI largely consists of an AVI interface and one or more analysis pipelines, all executed within the AVI Framework.

The AVI interface is used by users of GAVIP to configure the AVI (i.e. changing the input parameters to the analysis pipelines). The user interacts directly with the interface, but does not interact with the pipeline directly. Instead, jobs are created by the AVI and run when resources are made available. This is elaborated in later sections, but is useful to consider early on.

_images/avi_conceptual.png

AVI Templates

AVI templates are the images used to create AVI containers, and determine the libraries and runtimes available. Several AVI templates are available to help support different AVIs; e.g. a Fortran AVI. A list of initial AVI templates can be found below. Within each AVI template is the AVI Framework, which handles various operations including authentication, and pipeline management. An incomplete list of these templates are provided below.

Template ID Runtimes Description
ps_avi_python_2:1.0.0 Python 2.7 It includes all components required to execute either an AVI web-interface or AVI pipeline.
ps_avi_python_3:1.0.0 Python 3.5 It includes all components required to execute either an AVI web-interface or AVI pipeline.
ps_avi_python_2_java_8:1.0.0 Python 2.7 Java 8 An extension of ps_avi_python_2:1.0.0, which permits the execution of AVIs that have Java (8) dependencies.
ps_avi_python_3_java_8:1.0.0 Python 3.5 Java 8 An extension of ps_avi_python_3:1.0.0, which permits the execution of AVIs that have Java (8) dependencies.
ps_avi_python_2_fortran:1.0.0 Python 2.7 Fortran An extension of ps_avi_python_2:1.0.0, which permits the execution of AVIs that have Fortran dependencies.

AVI framework

The AVI framework is a Django application which is eventually augmented to include the AVI code. The framework provides:

  1. Authentication with the portal using Oauth.
  2. Standalone capabilities (where it runs locally for development processes)
  3. Pipeline database models which are used by analysis pipelines.
  4. The Anaconda software suite.

AVI code

The AVI code runs within the AVI framework; it is one “application” within the AVI Framework Django project. As such, it follows the Django design, consisting of urls, mapping to views, which interact with models, which are maintained in a database.

Beyond the design of a Django app, AVIs are designed to operate in two modes: the first mode provides a web-interface for scientists to interact with the AVI, and the second mode is used to perform complex analysis or other long running tasks. These form the front-end and back-end of an AVI.

The common layer between the front-end and back-end is the database, and the models used to represent data. Typically an AVI front-end is used to provide some interface which configures and creates a model instance (this stores parameters for the pipelines to operate). The back-end is then designed to use these parameters to perform some analysis task. The front-end is then used to visualize the results in some way. The AVI back-end and front-end are run separately when running within GAVIP (both are run simultaneously when the AVI runs in Standalone mode).

The AVI framework provides most of the infrastructure and logic for AVIs to operate (including decoupling the front-end and back-end). The developer needs only to provide their analysis pipeline, store the necessary attributes for it in a database model, and build an interface so that users can populate the model with values. The AVI pipeline and interface (back-end and front-end) are discussed further in the next two subsections.

AVI Pipelines (back-end)

Pipelines are a series of steps required to perform some analysis, with their dependencies and ordering entirely defined by code. The pipelines are executed using Luigi under the hood, and the classes used to define your pipeline extend the Luigi classes. As a result, the Luigi documentation can be used as a resource.

AVI pipelines are imported from the tasks.py file in your AVI. You may have pipelines in other files, but they must be imported and made available in this file.

Example pipelines are available in our example AVIs, and are also provided as part of the full AVI development tutorial.

AVI Interface (front-end)

The AVI interface is how users interact with your AVI. It is a web interface, and can include any static resources required, including JavaScript resources. The AVI framework includes the Django Rest Framework, which can be used to generate a REST-compliant API with rich web interfaces without writing HTML or JavaScript code.

It is recommended that developers begin by writing their analysis pipelines, creating the required models to store the pipeline parameters, and using the Django Rest Framework to create a usable web interface. Once the AVI is functional, custom web interfaces can be developed as desired, and can reuse the views provided by the Django Rest Framework.

Django tutorials, including the Django Rest Framework tutorials can be used as resource for AVI development, but it is important to remember that the AVI code provided by the developer plugs in to the AVI Framework, and is a Django application. In the Django tutorial, the AVI is analogous to the ‘polls’ application, and analogous to the ‘quickstart’ application in the Django Rest Framework tutorial.

Database

The AVI runs an SQLite3 database. The database is formed and managed using models, these are representations of data structures defined by Django. The database must be used to store the data required by the AVI pipelines (back-end) as the front-end and back-end are run in separate containers, at separate times. Each AVI is given a unique database for each user, which is accessed by the front-end and back-end instances. This is why the parameters for the AVI pipeline must be stored in a model.

Not all developers will be familiar with Django, or database-backed applications, and some of the steps involved in developing an AVI may be foreign; in particular, database migrations. During development, models are created to represent the data structures required by the AVI; models are classes with one or more fields. Occasionally, the database may need to be updated to reflect the AVI models. This process is known as creating and applying migrations - migrations are instructions generated by Django to synchronize the database structure with the AVI models. If the database is no longer synchronized with the AVI, errors will occur when using the AVI. Read the section on managing the AVI database for more information.

Internal ports and processes

Within an AVI container, there are a number of services running. Some of these are exposed to the host machine (or exposed to GAVIP when deployed).

Below is a list of ports exposed in the container and brief descriptions of what they serve.

10000:Django webapp port. The AVI web server listens on this port, and provides the AVI web interface (or front-end).
9002:Supervisor web interface. Supervisor is the technology used to run the different processes which an AVI needs to function. It exposes a web interface to control these processes (including start/stop/tail)
18888:Jupyter interface. The container provides an instance of the Jupyter notebook server, it is recommended that notebooks are used to test and develop analysis code for your AVI.

Internal processes

The following container processes are executed by supervisor, which runs when the AVI is in standalone mode:

gavip_avi:This is the Django framework that hosts the AVI app
worker-avi:Celery worker that performs asynchronous execution of AVI pipeline tasks
redis-server:The Redis server. Jobs are queued here for the worker (only when running in standalone mode).
prepare_avi:A process which prepares your AVI for execution when its deployed (including running migrations) or when the container instance is created
gavip_avi_jupyter:
 Jupyter notebook server

Example AVIs

A set of example AVIs are avilable on github at: https://github.com/parameterspace-ie/, these include:

example-avi-simple A basic AVI that demonstrates creating a functioning web interface and pipeline. Includes a REST API and generated interface.
example-avi-shared-data A sample AVI based on simple-avi which can be used to demonstrate the use of shared data-products
example-avi-samp Built on the simple_avi, it includes a SAMP client and monitor which connects to a local SAMP hub (note: one is provided by TopCat).
example-avi-gaiadr1 Example AVI that uses Gaia Data Release 1. Some of the processing has been incorporated from the notebooks at https://github.com/tboch/VO-access-GaiaDR1.
example-avi-resource Builds on the simple AVI to allow the user to define the RAM to be allocated for their pipeline. It includes a RAM resource slider. This is used to populate the ‘resources_ram_mb’ attribute of the AviJob model.
example-avi-multiple-pipelines Example AVI built on Simple-avi which demonstrates more complex analysis pipeline structures
example-avi-alerts Example AVI for GAVIP which gathers Gaia’s science alerts and plots the RA and dec of each object using mpld3.

These AVIs are released following Git-flow, so each release will update the Master branch. You can checkout historical versions of the AVIs if you wish.

Development Setup

To develop your AVI, you will need Docker installed, a text editor, and a terminal. It is recommended you also install the GAVIP Client.

Install Docker

Note: sudo may be required for any docker commands depending on installation. Alternatively, following release 1.12.0 of Docker, you can run Docker under a users namespace (running the containers as that user). For more information visit the Docker docs.

Follow the Docker installation guide here for your Operating System

First: verify docker is functional
  1. Check the Docker version::
    $ docker version
  2. Run the hello-world container::
    $ docker run hello-world
Second: test that you can run an ubuntu containers::
$ docker run -it ubuntu bash

Get a browser with a debugger built in

Most modern browsers will include a debugging tool (in Chrome, this can be accessed by pressing F12 on your keyboard). When developing your AVI interface, there is some web development involved, and so it is useful to have a debugger to inspect the web requests, or even to debug the interface itself.

Acquire the AVI template

First we will download an AVI template to create an AVI container to work with. This is a pre-emptive step, the actual setup is completed when Docker works!

  1. Login to the GAVIP container repository.
    $ docker login repositories.gavip.science Username/password: guest/gavip
  2. Pull the Python AVI template
    $ docker pull repositories.gavip.science/ps_avi_python_2:1.0.0

You now have the image on your machine, which contains the AVI framework.

A full tutorial is available to guide you through creating your first AVI, see Tutorial: AVI Development via Jupyter.

If you have already followed this tutorial, or want to start development using a working AVI, have a look at our example AVIs on GitHub.

AVI internals

In this section, we show the internal components of an AVI and describe how they are used. You can run an AVI on your machine using an AVI template and inspect these files for yourself.

Overview

First, lets recap on the primary technologies used in an AVI container.

Technology Usage
Python Development language
Django Development of pipeline models and web interfaces
Luigi Defining pipelines and their tasks
Celery Asynchronous AVI job (pipeline) execution
Supervisor Running AVI container processes simultaneously

GAVIP AVI framework structure

The AVI framework is embedded within AVI containers, and provides the infrastructure and much of the boilerplate functionality required for an AVI. This framework must be:

  • Able to support the queueing and execution of complex pipelines against the Gaia data archive;
  • Flexible so that developers can use tools and libraries not yet released;
  • Easy to use, as AVIs will not be developed if the process is too complex.

At a high level, the AVI framework is implemented as a web application using the Django web framework. The AVI created by developers forms one component or “application” which is run within the AVI framework. The framework handles pipeline management, GAVIP communications, and authentication with GAVIP. In addition, the framework supports standalone operation for development purposes.

The GAVIP AVI framework is provided as a Django project, with contains the following components (as Django applications):

avi:This is the component that AVI developers write. All other apps constitute the GAVIP AVI framework, and are provided in the container template images
avi_auth:Used to manage user authentication and authorisation for the AVI
base:An app containing base HTML layouts and models that can be used by all AVIs
connectors:Implementations of standard protocols (e.g. TAP+) that can be used in AVI pipelines (see subsection on protocols, etc)
gavip_admin:Place-holder for remote operator/developer administration of the AVI
gavip_avi:The default app (matching the Django project name). This includes settings for the AVI
messaging:Facilitates AVI-Portal communication
pipeline:Framework that executes AVI-defined pipelines for asynchronous task execution
plugins:Plugins provide bundled functionality, such as automatic job lists, to prevent unnecessary development
services:Ready-to-use utilities implementing the connectors

Example AVI structure

As shown in the GAVIP AVI framework structure, the avi app generated by AVI developers is one app within a gavip_avi framework project. The structure of a basic avi is shown below, with a brief description of some components. For more detail about the content of these files, it is recommended that you follow the Tutorial: AVI Development via Jupyter:

avi
├── __init__.py
├── templates
│   └── avi
│       └── index.html
├── models.py
├── tasks.py
├── urls.py
└── views.py
models.py:AVI models for persisting data
tasks.py:Analysis pipelines
urls.py:URL-view mapping
views.py:View functions to handle web requests
templates:HTML templates for your AVI interface

Protocols, connectors, services

Note: This section is described in further detail in the GAVIP SDD.

One of the more useful features of the AVI Framework is a suite of built-in utilities and functions which are commonly used in Astronomy applications. These are referred to as connectors and services, and can be used to easily acquire Gaia data as part of your pipeline, or even allow your AVI to interact with others using a SAMP interface.

Protocols:Protocols are taken from IVOA specifications, or internal GAVIP specifications.
Connectors:Small software modules implementing the protocol specification. For example, the TAP+ connector exposes functions for monitoring asynchronous jobs (connectors.tapquery).
Services:Services are implementations of connectors which can be easily added to your AVI pipeline. For example, the GacsQuery service provides a useful mechanism for downloading data from the GACS archive via the TAP+ connector, within an AVI pipeline task (services.gacs).

SAMP

When trying to use AVIs together, particularly in a dashboard view on the GAVIP portal, SAMP can be used as a message bus.

Currently SAMP is available within the AVI framework via samp.js, a javascript implementation of SAMP. The initialization of this library is handled by the framework automatically, creating a SAMP connector object in the browser. To include SAMP in your AVI interface, include base/samp.html. Refer to the example AVI samp_avi for a working example.

Installing dependencies

If your AVI requires packages that are not avaialable within the AVI template, they can be installed by providing AVI dependencies.

Overview

There are a number of AVI templates which support different runtimes including Python 2.7, Python 3.5, Java 7, Java 8, and Fortran (gfortran 4.8).

Below are details on how dependencies can be supplied to each of these AVI templates, such that the provided modules are installed on AVI startup. Within each of the below guides, the AVI directory refers to the AVI folder which contains (urls.py, models.py, tasks.py)

Template: ps_avi_python_2

Within the AVI directory provide a requirements file: “pip_requirements.txt”

When the AVI starts, it will locate this file and run:

pip install -r pip_requirements.txt

Template: ps_avi_python_3

Within the AVI directory provide a requirements file: “pip_requirements.txt”

When the AVI starts, it will locate this file and run:

pip install -r pip_requirements.txt

Template: ps_avi_python_2_java_8

Within the AVI directory provide a requirements file for any Python dependencies: “pip_requirements.txt” A Pom XML file can be provided for any Java dependencies: “pom.xml”

When the AVI starts, it will locate “pip_requirements.txt” file and run:

pip install -r pip_requirements.txt

If POM.xml is found it will run:

mvn dependency:resolve -f pom.xml

Template: ps_avi_python_3_java_8

Within the AVI directory provide a requirements file for any Python dependencies: “pip_requirements.txt” A Pom XML file can be provided for any Java dependencies: “pom.xml”

When the AVI starts, it will locate “pip_requirements.txt” file and run:

pip install -r pip_requirements.txt

If POM.xml is found it will run:

mvn dependency:resolve -f pom.xml

Template: ps_avi_python_2_fortran

Within the AVI directory provide a requirements file for any Python dependencies: “pip_requirements.txt” Fortran dependency installation will be added to the AVI templates soon.

When the AVI starts, it will locate “pip_requirements.txt” file and run:

pip install -r pip_requirements.txt

Models

AVIs use a database to persist information about job parameters.

Database technology

Databases within AVIs use a local SQLITE database. The requests to the database are extremely low in terms of what databases are designed to handle, as only one GAVIP user interacts with an AVI at a time (an AVI is deployed for each user on demand).

Database usage

Django handles most aspects of the database usage for you: https://docs.djangoproject.com/en/1.10/topics/db/

There are not many additional requirements imposed by GAVIP in how the database can be used.

Note

For your analysis pipeline to work, you must store the parameters in a model extending the AviJob model!

AviJob model

The AviJob model is an abstract model which AVI models will extend in order to leverage the AVI pipeline processing system.

The model is made available in your pipeline under the job_model field, so you can reference attributes from the model without needing to pass all other attributes through your pipeline.

Important AviJob attributes

The following attributes should be set in your model:

  • pipeline_task: The name of the analysis pipeline class to begin processing when the job model is created
  • resources_ram_mb: If not set, this defaults to a RAM allocation of 2GB

It may also be desirable to set expected_runtime, otherwise it is calculated by the median time of all tasks using this model.

The following are some of the attributes which can be retrieved from the model:

  • request.result_path: Used to record the final data product
  • request.output_path: Used for the directory of your data products (by default, this is generated from your job input parameters)
  • request.pipeline_state.state: Your job state
  • request.pipeline_state.progress: Your job progress in % (calculated as required/complete tasks)

There are many more fields available in the models, so it is highly recommended to refer to the AVI framework Sphinx documentation.

Pipelines

How AVI pipelines work

Pipelines are a crucial part of all AVIs. The AVI web application (hosted by Django) is single-threaded, so it should not be used to perform any processing - otherwise the browser client may timeout. The AVI framework allows developers to define pipelines within their AVI for long-running tasks, which are automatically queued and asynchronously executed.

Pipelines are implemented through a combination of Luigi and Celery:

  1. Luigi is used to define the AVI pipeline structure, including tasks and their dependencies.
  2. Celery is used to queue and manage asynchronous execution of an AVI pipeline.

How to create a pipeline

The Tutorial: AVI Development via Jupyter guides you through creating your first pipeline. It is recommended that developers refer to the Luigi documentation for in-depth pipeline documentation.

  1. Create a task class that derives pipeline.classes.AviTask (e.g. avi.tasks.ProcessData in simple_avi (Note: AviTask derives luigi.Task - pipelines are based on those defined in the luigi docs)
  2. Define required parameters at the start of the class, each one an instance of pipeline.classes.AviParameter.
  3. Define an output() function (specifies the task output file - luigi checks that this output file hasn’t already been created before running)
  4. Pipelines consist of chains of tasks that are dependent on other task execution. Task dependencies are specified using a requires() function.
    • Note: Instead of directly returning the pipeline class, requires() returns self.taskDependency(className, [args])
  5. Define the tasks run() function. This can contain any functionality, e.g. retrieving data from GACS-dev, analysing retrieved data etc..
    • Throughout the class, self.input() or self.output() to access input files from dependencies, or the output file of the task itself.

Task example

An example of a task with a single dependency is shown below:

class DummyTask(AviTask):
    """
    This is a sample task which has no dependencies.
    It only exists to further demonstrate dependency creation.
    """
    outputFile = AviParameter()

    def output(self):
        return AviLocalTarget(os.path.join(
            settings.OUTPUT_PATH, 'dummyData_%s.vot' % self.outputFile
        ))

    def run(self):
        with open(self.output().path, "w") as outFile:
            outFile.write("dummyStuff")


class DownloadData(svc_gacs.GacsQuery):
    """
    This task uses an AVI service, to obtain a data product from GACS.
    Notice that we do not define a 'run' function! It is defined by the
    service class which we extend.
    See : class : GacsQuery
    """

    query = AviParameter()
    output_file = AviParameter()

    def output(self):
        return AviLocalTarget(os.path.join(
            settings.OUTPUT_PATH, 'simulatedData_%s.vot' % self.output_file
      ))

    def requires(self):
        return self.task_dependency(DummyTask)

How to start a pipeline from a user request

To start a pipeline, you simply need to save a new instance of a model which extends the AviJob class. This is shown step-by-step in the Tutorial: AVI Development via Jupyter.

For example, the following code is a portion of the run_query() function from our Simple AVI example on GitHub. Note that a new instance of DemoModel is created in this function, initiating the pipeline job:

def run_query(request):
    outfile = request.POST.get("outfile")
    adql_query = request.POST.get("query")

    job = DemoModel.objects.create(
        query=adql_query,
        outputFile=outfile
    )

Running Java tools within an AVI pipeline

Although GAVIP is primarily suited to Python applications, the ps_avi_python_3_java_8 template container image facilitates the many existing DPAC tools that have been written in Java. Java tools can be executed within a local development/standalone container as follows:

  1. Use the ps_avi_java_8 template - this contains OpenJDK8, and the java executable will available in the PATH.
  2. Mount the Java tool directory into the container using the host location
  3. Execute the tool with java from a pipeline task, storing any outputs in /data/output as normal.

Java usage example

Create a container instance from the image, that mounts your AVI into the Django project located at /opt/gavip_avi/avi, and a Java tool (e.g. CU8 Ulysses) stored locally on the host machine:

sudo docker run \
    -e SETTINGS=settings.standalone \
    -v ~/my_first_avi/avi:/opt/gavip_avi/avi \
    -v ~/my_first_avi/data:/data \
    -v ~/dpac_tools/ulysses:/opt/ulysses \
    -p 10000:10000 \
    -t repositories.gavip.science/ps_avi_python_3_java_8:1.0.0 \
    /opt/conda/bin/supervisord

Note: Ulysses is mounted at /opt/ but this is not supported when the AVI is hosted in GAVIP. Future versions of this guide will detail how to import Java tools using the Shared Data functionality of GAVIP.

The Ulysses directory is now available within the container at /opt/ulysses, which means that the tool can be executed from AVI pipeline task run() functions, e.g.:

import shlex
import subprocess

...

cmd = 'java -jar dist/ulysses.jar -f "%s" -w %s -n %d -G %d -A %d -oversamp %d -conversion %d' % (
    spectra_input, wavelength_input,
    num_noisy_spectra,
    g_mag,
    extinction,
    oversampling,
    conversion
)

fp = subprocess.Popen(shlex.split(cmd), bufsize=-1, stdout=subprocess.PIPE, cwd='/opt/gavip_avi_java').stdout

...

Alternatives to using subprocess for Java execution from Python include direct Java classes access provided by Pyjnius. However, at the present time, it would appear that the subprocess approach will be most suitable - this approach is also used for Java tool execution by other Python-based analysis platforms, e.g. gensim.

Plugins

Plugins are features or utilities available within the AVI framework which can be included in your AVI if you wish. Below is a table summarising the currently available plugins (this table may expand in future). For each plugin there is a small subsection describing its usage.

Plugin Description
Job List Provides an interactive searchable table showing the status of your AVI jobs

Job List plugin

_images/plugin_job_list.png

Purpose

The job list plugin adds a job list based on any models which extend AviJob. It allows the user to:

  • View the result
  • View the data in their userspace
  • Download the file/directory
  • Share the data-product
  • Delete the job

The job progress is shown in a progressbar which also shows the active task within the analaysis pipeline.

Usage

Import the job_list urls to your urls.py

from plugins.urls import job_list_urls

Include the job_list urls in your urlpatterns (note, the namespace MUST be ‘job_list’)

url(r'^job_list/',
    include(job_list_urls,
    namespace='job_list')),

Ensure that you have a view for viewing the job result (it must accept a job_id argument)

url(r'^result/(?P<job_id>[0-9]+)/$',
    views.job_result,
    name='job_result')

Finally, in any of your templates, include the job_list plugin template

{% include "plugins/gavip_job_table.html" %}

Other notes

Refer to the AVI framework Sphinx documentation for more information regarding the functions and classes which provide the plugin functionality.

Interface development

One of the design choices made when developing GAVIP, was not to unnecessarily restrict the interface that an AVI can provide. The AVI-user interface is a web interface due to the GAVIP platform being hosted online, but the content that can be shown by an AVI is entirely at the developer’s discretion.

Suggested tools and resources

As the AVI interface can be very open-ended, there are some recommended tools or resources in this section to help a developer begin writing a complete AVI interface quickly.

Warning

Do not write a URL to begin with a forward slash (bad: /my_url/test/ good: my_url/test/). Doing so will stop the AVI from working correctly when it is deployed by GAVIP.

All example AVIs are already using the Django templating engine to build more dynamic web interfaces (e.g. embedding results in result pages). It is recommended that you review the Django template documentation if you have not already as it provides many useful features.

See also

The Django documentation on model forms: https://docs.djangoproject.com/en/1.10/ref/templates/builtins/

Django forms “”“”“”“”“”“”window.job_list_url = “{% url ‘avi:job_list:jobrequest_list’ %}”;

Once you have created your model in your AVI to store the parameters needed for your AVI pipeline, you need some way for developers to input their parameters. A developer can build this interface entirely themselves, but creating a URL to handle a POST request and using a django-forms to generate the HTML form dramatically reduces required effort. Refer to the “simple” AVI tutorial for a demonstration of this in action. This is a very common operation that can be dramatically simplified by using django forms.

See also

The Django documentation on model forms: https://docs.djangoproject.com/en/1.10/topics/forms/modelforms/

Bootstrap is included in the AVI framework along with Jquery (both are included automatically when extending “base/base.html” in your HTML template). There are lots of resources already provided in Bootstrap for more advanced interfaces.

See also

The Bootstrap documentation: http://getbootstrap.com/

See also

Working examples of useful bootstrap snippets: http://bootsnipp.com/

See also

One of many available bootstrap theme customizers: https://pikock.github.io/bootstrap-magic/

Rendering generated content

Often your AVI will create content such as images or even videos that you will want to show to the user.

In the AVI there are two types of resource that can be rendered: STATIC and MEDIA resources.

  • STATIC will include files such as Javascript or Stylesheets
  • MEDIA is used for user generated content

To support rendering generated content, the ‘/media/’ URL of your AVI maps to ‘/data/output’. So if you have an image /data/output/sample/myimage.png, you can use /media/sample/myimage.png to render it in your AVI.

As your AVI will be run on some subpath of the GAVIP platform (i.e. http://GAVIPURL/some_unique_identifier/avi/) it is recommended that you use the following code to ensure that the a Media URL is correctly generated (the load static tag is only required once in your HTML template):

{% load static %}
<img src="{% get_media_prefix %}/some_directory/sample.png"></img>

Embedding AVI URLs in your template

To embed the correct path to different URLs of your AVI, you can use the reverse tag in your HTML template.

For example, in a HTML form we can embed the URL to submit the form (run_query from our AVI tutorial):

<form enctype="multipart/form-data" action="{% url 'avi:run_query' %}" method="post">

Or we can store a URL as a Javascript variable:

window.query_submit_url = "{% url 'avi:run_query' %}";

Note

The string we are using consists of ‘avi:<name of view>’, so ‘run_query’ is the name used in urls.py which happens to be the same as the URL path in this case.

Warning

All URLs provided by your AVI are within the ‘avi’ namespace, so to access them, you must prepend ‘avi:’ when looking them up.

Submitting an AVI to GAVIP

Adding your AVI to GAVIP is a 3 step process. Before adding your AVI to GAVIP, ensure that your AVI code is hosted in a Git repository (public or private repositories are suitable).

If you wish, you can request a private repository to be created for you by ESAC. If you have your own private repository, ensure that GAVIP will be permitted to read from it by adding GAVIPs public key:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwaJcxb7/3ijYlkX/jmpSO6wL8R1kYO9ZzELDuV3nmRfxIlrq8Texei9O4jr83Tfw9STbokgGraR6uEr5JoPnbKgQu1KDLcyuWJO1UZu3UG6Sy4Ewjuhpt4KGhdPIOhpPamXg9wjLEw/YZbtUAnbfTZDvyHe0E5awOiSIHx5hAD9uouxUadLbQBHALuktG6gIhQWKKsY8IxLjEoaVrbYMZZADZW4jJ8cZ/vneY3C6CxKcPfLFWVqTJLZsbRpkEZf7rvLZrjkNa7NvmHHWxoGb/WYfZIJqRsrRkhH9Fyy1gBDLrXdx+ojjPM3h9W1fRsg8gdvB3JWl7wRc9QYD6QUOl gavip

Project and Releases

Once you have ensured your AVI code is hosted in a Git repository, you simply need to login to GAVIP and create an AVI project, followed by an AVI release, then finally initiate a build of your AVI.

AVI Project creation

An AVI project is a record which specifies the detail of your AVI’s Git repository, and some metadata such as AVI title and author. A screenshot of the standard portal interface seen when adding a project is provided below.

_images/projects.png

AVI Release creation

Once an AVI project is created, a release must be made which specifies what version of the code to make available. An AVI release is a record consisting of a Git hash (a unique identifier for a historic state of your Git code), a version tag (can be a number, or a phrase), and a flag indicating whether the release is to be made public or not. If the release is not made public, it will only be made available to the AVI developer rather than all users of the platform.

_images/release.png

AVI Build creation

Once an AVI release has been created you can create your AVI build, which then becomes one of the AVIs in the AVI catalog. To build your AVI, navigate to the “Manage Builds” section under the “Developer” view in GAVIP, and select your release. The release details will popup, and you can initiate a build; while building, you can click the “Reload” button to view the latest build status including build log.

_images/avibuild_1.png _images/avibuild_2.png

When you tell GAVIP to build your AVI, your Git code is downloaded and used to create a unique container image. The container image is stored in the GAVIP repository The build process is outlined below:

  1. AVI code is checked out and copied into a unique AVI image
  2. Any specified software pre-requisites are installed
  3. Tests are run against the code (if these fail, the AVI is not made available)
  4. The REST API is examined and saved

When this process completes, your AVI is available within the GAVIP AVI Catalog.

AVI tests

AVI tests are executed by GAVIP during the build. This is intended as a sanity test and not a complete test of your AVI. Full AVI testing should be performed while in development, and tests should pass before a public AVI is built.

If you would like to skip some tests while in the AVI build (e.g. if they require downloaded files, or mounted content which is not available during the build), you can use a unittest skip decorator and testing for the ‘IN_GAVIP_BUILD’ attribute in settings. This attribute is only True during an AVI build. An example is provided below, refer to the unittest Documentation

UNITTESTING_MODE = not getattr(settings, 'IN_GAVIP_BUILD', False)

@skipUnless(UNITTESTING_MODE, "Executed only in unit testing mode")
def test_sample_skipped_test(self):
    """This will be skipped during an AVI build"""
    pass

Caveats

AVIs log via Syslog over UDP. As such, some logs may exceed the maximum content length supported, and may result in an error while logging (this has been observed when the log message is several thousand characters long).

URLs in your AVI must never begin with a leading forward slash (otherwise it wont work when deployed by GAVIP).

Contact

Issues relevant for local/standalone development

  1. All container process log files are stored in /data/logs - these should be checked when diagnosing any problems that occur.
  2. If models in avi.models are modified, where avi is mounted into an AVI container, it may be required to ssh to the container to execute standard Django migration commands from /opt/gavip_avi (or delete the database and restart AVI!).
  3. If avi.tasks is updated, the AVI worker process will need to be manually restarted. This is achieved by either of the following steps:
    • Use the GAVIP Client command (gavip_client development restart_worker)
    • Open the the supervisord web-interface at http://<container IP>:9002 and restart worker-avi.
    • Exec into the container docker exec -it <container> bash, and execute $ supervisorctl restart worker-avi