Design and anatomy of GWCelery¶
Several online gravitational-wave transient search pipelines (currently Gstlal, PyCBC, cWB, and oLIB) upload candidates in real time to GraceDB, the central database and web portal for low-latency LIGO/Virgo analyses. Whenever an event is uploaded or altered, GraceDB pushes machine-readable notifications through LVAlert, a pubsub system based on XMPP.
The business logic for selecting and sending alerts to astronomers resides not in GraceDB itself but in GWCelery. The role of GWCelery in the LIGO/Virgo alert infrastructure is to drive the workflow of aggregating and annotating gravitational-wave candidates and sending GCN Notices to astronomers.
GWCelery interacts with GraceDB by listening for LVAlert messages and making REST API requests through the GraceDB client. GWCelery interacts with GCN by listening for and sending GCN Notices using the Comet VOEvent broker.
The major subsystems of GWCelery are:
the LVAlert listener
the GraceDB client
the GCN listener
the GCN broker
the Superevent Manager, which clusters and merges related candidates into “superevents”
the External Trigger Manager, which correlates gravitational-wave events with GRB, neutrino, and supernova events
the Orchestrator, which executes the per-event annotation workflow
Below is a diagram illustrating the conceptual relationships of these subsystems. Nodes in the graph are hyperlinks to the relevant API documentation.
Scheduler for periodic tasks (the Celery equivalent of cron jobs). For more information, see Periodic Tasks in the Celery manual.
Monitoring Console (optional)
You can optionally run Flower, a web monitoring console for Celery.
A Celery worker that has been configured to accept only computationally intensive tasks that use OpenMP parallelism. To route a task to the OpenMP worker, pass the keyword argument
@app.taskdecorator when you declare it.
There are two tasks that run in the OpenMP queue:
A Celery worker that is dedicated to serially process triggers from low latency pipelines and create/modify superevents in GraceDB. There is only one task that runs on the Superevent queue:
External Trigger Worker
A Celery worker that is dedicated to serially process external triggers from GRB alerts received from Fermi, Swift, Integral, Agile MCAL and neutrino alerts received from SNEWS and create/modify external trigger events in GraceDB:
A Celery worker that is dedicated to sending and receiving VOEvents. It runs an embedded instance of the Comet VOEvent broker, which is started and stopped using a set of custom Celery bootsteps. Note that the VOEvent worker must be started with the
--pool=solooption so that tasks are executed in the same Python process that is running the VOEvent broker.
A Celery worker that accepts all other tasks.
Flask Web Application
A web application that provides forms to manually initiate certain tasks, including sending an update alert or creating a mock event.
GWCelery has a few long-running tasks that do not return because they have to
keep open a persistent connection with some external service. These tasks are
These tasks run inside the general-purpose worker process described above, and are automatically started (and restarted as necessary) by Celery Beat.
A recurring pattern in GWCelery is that an eternal task listens continuously to a remote connection, receives packets of data over that connection, and dispatches further handling to other tasks based on packet type.
A decorator is provided to register a function as a Celery task and also plug it in as a handler for one or more packet types. This pattern is used for both GCN notices and LVAlert message handlers.
GCN notice handler tasks are declared using the
import lxml.etree from gwcelery.tasks import gcn @gcn.handler(gcn.NoticeType.FERMI_GBM_GND_POS, gcn.NoticeType.FERMI_GBM_FIN_POS) def handle_fermi(payload): root = lxml.etree.fromstring(payload) # do work here...
LVAlert message handler tasks are declared using the
from gwcelery.tasks import lvalert @lvalert.handler('cbc_gstlal', 'cbc_spiir', 'cbc_pycbc', 'cbc_mbtaonline') def handle_cbc(alert): # do work here...