Survey6

Ipv6 is the internet’s future, and it necessitated a more scalable survey tool to comprehend how routing and DNS function. The purpose of this project is to create an IPv6 listener that will passively collect IPv6 traffic data as a passive data collection tool for cyber security research.

survey6

Join the chat at https://gitter.im/web-telescope/survey6

ipv6 survey Tool (survey6)

Brief explanation:

Ipv6 is the internet’s future, and it necessitated a more scalable survey tool to comprehend how routing and DNS function. The purpose of this project is to create an IPv6 listener that will passively collect IPv6 traffic data as a passive data collection tool for cyber security research.

Goals:

  • Develop a Linux network probe to intercept ipv6 traffic
  • To centralize the data being intercepted by the probe, develop a geo-distributed grid application that integrated with the probe.

Survey6 tool has three main components, follow describes each component, its functionalities and setup.

  1. Probe
  2. C&C Server
  3. Data Aggregator

chart2 You can find setup and installation guide for each of the modules in their respective module directories and they are linked below too. ***

Probe

Intercept and collect all the ipv6 traffic (regardless of the protocol) therefore libpcap is preferable. probe binaries must run as a service of the operating system (OS could be Linux host). Moreover, the probe cast a heartbeat to the C&C server for its heath checks mechanisms. This must be implemented using gRPC. Probe identifies ideal states of the host network interface and uses those time windows to send the collected ipv6 pcap. For this, data must be annotated with meta-information for aggregation purposes (meta information could be discussed)

Probe CLI

Probe CLI is a sub-component of Probe that allows starting (passing the registering string from C&C server, eg- How you add new nodes to Kubernetes), suspending the probe’s execution in the host machine.

Stack

Python, GRPC, Scapy

Module Info, Setup \& Installation Guide

Demo

C&C server should have the Probe registering mechanism. And it listens to registered Prob’s heartbeats. WebUI shows the active states of Probes in a list view.

Stack

Python, gRPC, Redis queue, SQLite, Flask

Module Info, Setup \& Installation Guide

Demo Video


Data Aggregator.

Data Aggregator is a series of scheduled Apache Airflow Dags

  • DAG1 - Use dpkt or scapy to parse the pcap files along with metadata.
  • DAG2 - Spark Jobs to clean and aggregate the data and write to Parquet.
  • DAG3 - Error handling and cleansing temp data.

Stack

Python, AirFlow, Spark, PySpark, scapy


Contributing

We are glad that you are willing to contribute. Tasks to be taken up next are:

  1. Fix issue that fails one testcase - C&C Server
  2. Heartbeat and health status check - both modules : Probe and C&C Server
  3. Sending packets from probe and receiving packets at server - both modules : Probe and C&C Server
  4. Create a debian package for the probe once it has significant development. [For this one could refer packaging procedure of C&C Server: here and here] - Probe
  5. Display status of connected and registered probes - C&C server

Stretch goal:

  1. Hosting the debian packages

Please do follow contributing guidelines to help the maintainers. Thank you for your wonderful cooperation!