In-network computing: solutions for graph analytics

The constant stream of data generated by people, software systems and the physical world has three distinguishing characteristics: it is very large, it is very varied and it accumulates very fast. Information is also more accessible than ever before. For example, in many applications (e.g. phone calls and social media), data that record interactions between people, systems and the environment are available for analysis. The goal of this project was to conceive, develop and validate novel techniques and approaches for exploiting these data. The project tackled three main challenges: how to store and manage the data; how to ask questions about the data that provide useful answers; and how to “see” changes in the data over time. Past research efforts have focused on data related to static relationships between entities (people, places and things). But data change over time, and we need to understand the trends. In addition, the amount of data recorded is constantly growing, which makes efficiently storing and processing it a significant challenge.

Past research efforts have focused on data related to static relationships between entities (people, places and things). But data change over time, and we need to understand the trends. In addition, the amount of data recorded is constantly growing, which makes efficiently storing and processing it a significant challenge.

Our project intended to tackle three main challenges related to processing and visualising data about interactions. First, we wanted to develop the computer system infrastructure needed to store and manage the data. Second, we wanted to create a specialised computer language that allows users to ask questions about the data. Finally, we wanted to design a method of visualising data that allows users to see how the data change over time.

Given the wide variety of applications for data that record interactions, and the need to understand and gain insights from those data, we expected the successful completion of this project to have a broad impact on both research and industry. We have already been working with representatives from telecommunications companies and internet service providers who would like to use the tools we develop, as well as database vendors who are interested in incorporating our techniques into their products.

The initial project proposed exploring data structures for doing efficient transformations on graph data. However, during the course of our study, we shifted in a research direction that was not anticipated in the original project plan: exploring and evaluating uses of in-network computing. In-network computing is a special form of hardware acceleration in which programmable network hardware is used to accelerate or offload application logic.

This shift proved to be quite fruitful, as we had a number of strong research results, including publications at well-respected scientific venues, including SOSR, HotNets, NSDI, and CoNEXT. Our work was able to demonstrate that in-network computing can achieve significant performance improvements for distributed applications, sometimes up to five orders of magnitude increases in throughput. One highlight was that our publication on Packet Subscriptions, which accelerated publish-subscribe communication patterns, received the Best Paper award at CoNEXT 2020.

Exploratory Visual Analytics for Interaction Graphs