{"id":2001,"date":"2025-07-13T22:34:24","date_gmt":"2025-07-13T22:34:24","guid":{"rendered":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/?p=2001"},"modified":"2025-07-14T00:00:03","modified_gmt":"2025-07-14T00:00:03","slug":"ndpi-network-traffic-analysis","status":"publish","type":"post","link":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/?p=2001","title":{"rendered":"nDPI Network Traffic Analysis"},"content":{"rendered":"\n<figure class=\"wp-block-audio\"><audio controls src=\"http:\/\/172-234-197-23.ip.linodeusercontent.com\/wp-content\/uploads\/2025\/07\/nDPI-Network-Traffic-Inspection.mp3\"><\/audio><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Podcast: explore FCC Enforcement Applications of nDPI<\/h2>\n\n\n\n<p>Welcome to another episode of our podcast, where we explore the tools and code that power modern network analysis! Today, we\u2019re taking a close look at\u00a0<code>ndpiSimpleIntegration.c<\/code>, a comprehensive example from the nDPI project\u2014a popular open-source deep packet inspection toolkit. ndpiSimpleIntegration.c \u2013 Network Traffic Analysis in Action<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What is nDPI?<\/h3>\n\n\n\n<p>nDPI is a library designed for network traffic classification, protocol detection, and traffic analysis. It\u2019s widely used in network monitoring, security, and research. The project is maintained by ntop.org and is known for its extensibility and performance.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Spotlight: ndpiSimpleIntegration.c<\/h3>\n\n\n\n<p>The&nbsp;<code>ndpiSimpleIntegration.c<\/code>&nbsp;file is more than just a sample\u2014it\u2019s a full-featured integration example that demonstrates how to use nDPI for real-world packet capture and analysis. Here\u2019s what makes it interesting:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Multi-threaded Packet Processing:<\/strong>&nbsp;The code supports multiple reader threads, each handling its own packet capture workflow. This design allows for efficient use of multi-core systems and high-throughput analysis.<\/li>\n\n\n\n<li><strong>Protocol Detection:<\/strong>&nbsp;Leveraging nDPI\u2019s detection engine, the example identifies application-layer protocols (like HTTP, TLS, and more) in real time.<\/li>\n\n\n\n<li><strong>Flow Tracking:<\/strong>&nbsp;The code maintains active and idle flows, tracking their state, timeouts, and protocol classification.<\/li>\n\n\n\n<li><strong>PCAP Integration:<\/strong>&nbsp;It can process both live network interfaces and offline PCAP files, making it versatile for both monitoring and forensic analysis.<\/li>\n\n\n\n<li><strong>Extensive Logging:<\/strong>&nbsp;With verbose output enabled, you get detailed insights into each packet, flow, and protocol detected.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Why Should You Care?<\/h3>\n\n\n\n<p>If you\u2019re interested in network security, traffic engineering, or just want to understand what\u2019s happening on your network, nDPI and this example are invaluable. The code is well-structured, heavily commented, and demonstrates best practices for integrating DPI into your own tools.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Key Takeaways from the Code<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Initialization:<\/strong>&nbsp;The workflow sets up packet capture, applies BPF filters, and initializes the nDPI detection module.<\/li>\n\n\n\n<li><strong>Flow Management:<\/strong>&nbsp;Flows are hashed and distributed across threads, with logic to handle timeouts and resource cleanup.<\/li>\n\n\n\n<li><strong>Protocol Guessing:<\/strong>&nbsp;Even when full detection isn\u2019t possible, the code attempts to guess the protocol, providing useful insights.<\/li>\n\n\n\n<li><strong>Signal Handling:<\/strong>&nbsp;Graceful shutdown is supported, ensuring resources are freed and statistics are printed.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Final Thoughts<\/h3>\n\n\n\n<p>Whether you\u2019re a developer, researcher, or network enthusiast, exploring&nbsp;<code>ndpiSimpleIntegration.c<\/code>&nbsp;is a great way to learn about deep packet inspection and high-performance network analysis. Check out the nDPI project on GitHub, and don\u2019t forget to listen to our full episode for a line-by-line walkthrough and expert commentary!<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p>The nDPI packet and flow analysis described in the sources allows for the analysis of various unencrypted network packet information across different layers, as well as derived flow-specific details.<\/p>\n\n\n\n<p>Here&#8217;s a breakdown of the unencrypted information that can be analyzed:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ethernet Layer (Layer 2) Information:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Ethernet Type (<code>h_proto<\/code>):<\/strong> The protocol type encapsulated in the Ethernet frame, such as <code>ETH_P_IP<\/code> (0x0800) for IPv4, <code>ETH_P_IPV6<\/code> (0x86DD) for IPv6, and <code>ETH_P_ARP<\/code> (0x0806) for ARP. Packets with unknown or non-IP\/Ethernet datalink types are skipped.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>IP Layer (Layer 3) Information:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Layer 3 Type (<code>l3_type<\/code>):<\/strong> Indicates whether the packet is IPv4 (<code>L3_IP<\/code>) or IPv6 (<code>L3_IP6<\/code>).<\/li>\n\n\n\n<li><strong>Source IP Address:<\/strong> <code>src<\/code> for IPv4 and <code>src<\/code> for IPv6, allowing conversion to human-readable strings.<\/li>\n\n\n\n<li><strong>Destination IP Address:<\/strong> <code>dst<\/code> for IPv4 and <code>dst<\/code> for IPv6, also convertible to human-readable strings.<\/li>\n\n\n\n<li><strong>IP Protocol (<code>l4_protocol<\/code>):<\/strong> The protocol number of the next layer (Layer 4), such as TCP, UDP, ICMP, ICMPv6, or IP options.<\/li>\n\n\n\n<li><strong>IP Size (<code>ip_size<\/code>):<\/strong> The size of the IP packet, excluding the Ethernet header.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Transport Layer (Layer 4) Information:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Layer 4 Protocol (<code>l4_protocol<\/code>):<\/strong> Identifies the transport protocol, specifically differentiating between <strong>TCP<\/strong> (<code>IPPROTO_TCP<\/code>) and <strong>UDP<\/strong> (<code>IPPROTO_UDP<\/code>) for further parsing. It also identifies ICMP, ICMPv6, and IP options.<\/li>\n\n\n\n<li><strong>Source Port (<code>src_port<\/code>):<\/strong> The port number of the source application.<\/li>\n\n\n\n<li><strong>Destination Port (<code>dst_port<\/code>):<\/strong> The port number of the destination application.<\/li>\n\n\n\n<li><strong>TCP Flags (for TCP packets):<\/strong>\n<ul class=\"wp-block-list\">\n<li><code>syn<\/code>: Indicates if the SYN flag is set.<\/li>\n\n\n\n<li><code>fin<\/code>: Indicates if the FIN flag is set.<\/li>\n\n\n\n<li><code>ack<\/code>: Indicates if the ACK flag is set.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>is_midstream_flow<\/code>:<\/strong> A flag indicating if a TCP flow started mid-stream (SYN flag not seen).<\/li>\n\n\n\n<li><strong><code>flow_fin_ack_seen<\/code>:<\/strong> A flag indicating if a TCP FIN+ACK packet has been observed for the flow.<\/li>\n\n\n\n<li><strong><code>flow_ack_seen<\/code>:<\/strong> A flag indicating if a TCP ACK packet has been observed for the flow.<\/li>\n\n\n\n<li><strong><code>total_l4_data_len<\/code>:<\/strong> The cumulative length of Layer 4 data processed for a flow.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Derived and Flow-Specific Information:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong><code>flow_id<\/code>:<\/strong> A unique identifier assigned to each detected flow.<\/li>\n\n\n\n<li><strong><code>packets_processed<\/code>:<\/strong> The number of packets processed for a specific flow.<\/li>\n\n\n\n<li><strong>Timestamps:<\/strong>\n<ul class=\"wp-block-list\">\n<li><code>first_seen<\/code>: The timestamp (in milliseconds) when the first packet of the flow was observed.<\/li>\n\n\n\n<li><code>last_seen<\/code>: The timestamp (in milliseconds) when the most recent packet of the flow was observed.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><code>hashval<\/code>:<\/strong> A calculated hash value for the flow, used for efficient lookup in data structures.<\/li>\n\n\n\n<li><strong><code>detected_l7_protocol<\/code>:<\/strong> The detected Layer 7 (application) protocol, including its master protocol, application protocol, and category. This is determined through deep packet inspection by the <code>ndpi_detection_process_packet<\/code> function.<\/li>\n\n\n\n<li><strong><code>guessed_protocol<\/code>:<\/strong> A protocol guessed as a last resort if a definitive detection isn&#8217;t possible after a certain number of packets.<\/li>\n\n\n\n<li><strong><code>detection_completed<\/code>:<\/strong> A flag indicating whether Layer 7 protocol detection has been finalized for the flow.<\/li>\n\n\n\n<li><strong>TLS-specific Details (extracted from unencrypted TLS handshake messages):<\/strong>\n<ul class=\"wp-block-list\">\n<li><code>tls_client_hello_seen<\/code>: Flag indicating if a TLS Client Hello message has been processed.<\/li>\n\n\n\n<li><code>tls_server_hello_seen<\/code>: Flag indicating if a TLS Server Hello message (containing certificate info) has been processed.<\/li>\n\n\n\n<li><strong>SSL\/TLS Version:<\/strong> The version of SSL\/TLS used, e.g., TLSv1.2, TLSv1.3.<\/li>\n\n\n\n<li><strong>Server Name Indication (SNI):<\/strong> The hostname the client is trying to connect to (<code>host_server_name<\/code>).<\/li>\n\n\n\n<li><strong>Advertised ALPNs:<\/strong> Application-Layer Protocol Negotiation values advertised by the client.<\/li>\n\n\n\n<li><strong>Certificate Common Name(s):<\/strong> Hostnames listed in the server&#8217;s certificate (<code>server_names<\/code>).<\/li>\n\n\n\n<li><strong>Certificate Issuer DN:<\/strong> The distinguished name of the certificate issuer (<code>issuerDN<\/code>).<\/li>\n\n\n\n<li><strong>Certificate Subject DN:<\/strong> The distinguished name of the certificate subject (<code>subjectDN<\/code>).<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Flow Information String:<\/strong> A human-readable string containing various details about the flow, obtained via <code>ndpi_get_flow_info<\/code>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Let&#8217;s delve deeper into how nDPI analyzes unencrypted network packet information, building upon our previous discussion. The process involves a multi-layered approach, from basic packet parsing to sophisticated flow tracking and application-layer protocol detection.<\/p>\n\n\n\n<p>Here&#8217;s an enhanced understanding of the analysis capabilities based on the sources:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Packet Capture and Initial Filtering<\/strong>\n<ul class=\"wp-block-list\">\n<li>The <code>nDPI_workflow<\/code> structure manages the packet capture process, using <code>pcap_open_live<\/code> for live capture or <code>pcap_open_offline<\/code> for reading from a PCAP file.<\/li>\n\n\n\n<li>A Berkeley Packet Filter (BPF) is applied, specifically configured to only capture &#8220;ip or ip6&#8221; packets, meaning it focuses solely on IPv4 and IPv6 traffic at Layer 3. Packets with other datalink types or those too short to contain essential headers (Ethernet, IP) are explicitly skipped.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Layer-by-Layer Parsing (<code>ndpi_process_packet<\/code>)<\/strong>\n<ul class=\"wp-block-list\">\n<li>Each captured packet is passed to the <code>ndpi_process_packet<\/code> function for analysis.<\/li>\n\n\n\n<li><strong>Datalink Layer (Layer 2):<\/strong> The function first determines the datalink type (e.g., DLT_EN10MB for Ethernet) to correctly interpret the packet&#8217;s structure. It then identifies the Ethernet type (<code>h_proto<\/code>), such as <code>ETH_P_IP<\/code> (0x0800) for IPv4 or <code>ETH_P_IPV6<\/code> (0x86DD) for IPv6. ARP packets (<code>ETH_P_ARP<\/code>) are explicitly skipped.<\/li>\n\n\n\n<li><strong>Network Layer (Layer 3):<\/strong> The packet is then parsed as an IPv4 (<code>ndpi_iphdr<\/code>) or IPv6 (<code>ndpi_ipv6hdr<\/code>) packet. Key information extracted includes:\n<ul class=\"wp-block-list\">\n<li><strong>Source and Destination IP Addresses:<\/strong> <code>saddr<\/code>\/<code>daddr<\/code> for IPv4 and <code>ip6_src<\/code>\/<code>ip6_dst<\/code> for IPv6, which are stored in the <code>ip_tuple<\/code> union within the <code>nDPI_flow_info<\/code> structure. These are later converted to human-readable strings for output.<\/li>\n\n\n\n<li><strong>IP Size:<\/strong> The total length of the IP packet, excluding the Ethernet header.<\/li>\n\n\n\n<li><strong>Layer 4 Protocol:<\/strong> The <code>ndpi_detection_get_l4<\/code> function is called to identify the next layer&#8217;s protocol (e.g., TCP, UDP, ICMP, ICMPv6) and locate its starting pointer (<code>l4_ptr<\/code>) and length (<code>l4_len<\/code>).<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Transport Layer (Layer 4):<\/strong> Based on the identified Layer 4 protocol:\n<ul class=\"wp-block-list\">\n<li>For <strong>TCP<\/strong> (<code>IPPROTO_TCP<\/code>), the source (<code>source<\/code>) and destination (<code>dest<\/code>) ports are extracted. Importantly, TCP flags like <strong>SYN, FIN, and ACK<\/strong> are examined. These flags inform the <code>is_midstream_flow<\/code> (if SYN was not seen), <code>flow_fin_ack_seen<\/code> (if FIN+ACK was observed), and <code>flow_ack_seen<\/code> flags in the <code>nDPI_flow_info<\/code> structure, which are critical for understanding the flow&#8217;s state and lifecycle.<\/li>\n\n\n\n<li>For <strong>UDP<\/strong> (<code>IPPROTO_UDP<\/code>), the source (<code>source<\/code>) and destination (<code>dest<\/code>) ports are similarly extracted.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Flow Identification and Management<\/strong>\n<ul class=\"wp-block-list\">\n<li>nDPI organizes packets into &#8220;flows,&#8221; represented by the <code>nDPI_flow_info<\/code> structure.<\/li>\n\n\n\n<li><strong>Flow Hashing:<\/strong> A <code>hashval<\/code> is calculated for each packet based on its Layer 4 protocol, source\/destination IP addresses, and source\/destination ports. This hash helps in efficiently locating or inserting flows into a b-tree data structure (<code>ndpi_tfind<\/code>, <code>ndpi_tsearch<\/code>).<\/li>\n\n\n\n<li><strong>Symmetry Handling:<\/strong> Since network flows can be bidirectional, if a flow isn&#8217;t found using the original source\/destination, nDPI attempts to find it by swapping the source and destination IP addresses and ports. This ensures that both directions of a single conversation map to the same flow entry.<\/li>\n\n\n\n<li><strong>Flow State Tracking:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Each flow is assigned a unique <code>flow_id<\/code>.<\/li>\n\n\n\n<li><code>packets_processed<\/code> tracks the number of packets belonging to a flow.<\/li>\n\n\n\n<li><code>total_l4_data_len<\/code> accumulates the total Layer 4 data length for the flow.<\/li>\n\n\n\n<li><code>first_seen<\/code> and <code>last_seen<\/code> timestamps record the beginning and end of observed activity for a flow, crucial for timeout handling.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Idle Flow Management:<\/strong> A <code>check_for_idle_flows<\/code> mechanism periodically scans active flows. Flows are marked as idle and prepared for freeing if a TCP FIN+ACK sequence is observed or if they exceed a <code>MAX_IDLE_TIME<\/code> (300 seconds or 5 minutes) since their <code>last_seen<\/code> timestamp.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Application Layer Protocol Detection (Layer 7 DPI)<\/strong>\n<ul class=\"wp-block-list\">\n<li>The core of nDPI&#8217;s analysis is the <code>ndpi_detection_process_packet<\/code> function, which performs deep packet inspection to identify the application-layer protocol (<code>detected_l7_protocol<\/code>). This protocol is categorized by its <code>master_protocol<\/code>, <code>app_protocol<\/code>, and <code>category<\/code> (e.g., NDPI_PROTOCOL_HTTP, NDPI_PROTOCOL_FACEBOOK, etc.).<\/li>\n\n\n\n<li><strong>Detection Completion:<\/strong> Once a protocol is definitively detected, the <code>detection_completed<\/code> flag is set, and flow information (including the detected protocol name and category) is printed.<\/li>\n\n\n\n<li><strong>Protocol Guessing:<\/strong> If a protocol cannot be conclusively identified after a certain number of packets (e.g., <code>0xFF<\/code> packets), <code>ndpi_detection_giveup<\/code> attempts to make a <code>guessed_protocol<\/code> based on heuristics, providing a &#8220;last chance&#8221; classification.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>TLS Handshake Information Extraction (Unencrypted Parts)<\/strong>\n<ul class=\"wp-block-list\">\n<li>Even for encrypted traffic like TLS, the initial handshake messages contain unencrypted metadata that nDPI can analyze.<\/li>\n\n\n\n<li><strong>Client Hello:<\/strong> When a TLS Client Hello message is processed, nDPI extracts and reports the <strong>SSL\/TLS version<\/strong> (<code>ssl_version<\/code>), the <strong>Server Name Indication (SNI)<\/strong> which is the hostname the client is trying to connect to (<code>host_server_name<\/code>), and <strong>Advertised ALPNs<\/strong> (Application-Layer Protocol Negotiation values). These are key pieces of information visible even without decrypting the payload.<\/li>\n\n\n\n<li><strong>Server Hello (Certificate Information):<\/strong> Upon processing the TLS Server Hello, if certificate information is available, nDPI can report the <strong>SSL\/TLS version<\/strong>, <strong>Common Name(s)<\/strong> from the server&#8217;s certificate (<code>server_names<\/code>), the <strong>Issuer Distinguished Name (DN)<\/strong> (<code>issuerDN<\/code>), and the <strong>Subject Distinguished Name (DN)<\/strong> (<code>subjectDN<\/code>). This allows identification of the server and certificate authority.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>nDPI provides a comprehensive framework for analyzing unencrypted network packet information by meticulously parsing each layer, managing flows based on extracted tuples, intelligently identifying application protocols, and extracting valuable metadata even from the unencrypted portions of otherwise encrypted protocols like TLS. This granular analysis is crucial for network monitoring, security, and traffic shaping applications.<\/p>\n\n\n\n<p>nDPI&#8217;s capabilities in analyzing unencrypted network packet information can indirectly, yet significantly, assist the FCC in its Spectrum Enforcement efforts. While nDPI does not directly analyze radio frequencies or physical layer characteristics, it provides deep visibility into the <strong>types of traffic and applications utilizing the network infrastructure<\/strong> that operates over the radio spectrum. This allows for the identification of potential misuse, unauthorized services, or unusual patterns that might violate spectrum regulations or contribute to interference.<\/p>\n\n\n\n<p>Here&#8217;s how nDPI can help:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Identification of Application Protocols and Services:<\/strong>\n<ul class=\"wp-block-list\">\n<li>nDPI can accurately detect the <strong>Layer 7 (application) protocol (<code>detected_l7_protocol<\/code>)<\/strong> for each network flow, including its master protocol, application protocol, and category. This is critical because specific spectrum allocations are often granted for particular types of services (e.g., licensed radio for private voice, broadband data).<\/li>\n\n\n\n<li>If, for example, a licensed frequency band is authorized only for specific data applications, nDPI could identify if <strong>unauthorized or unapproved applications<\/strong> (like large-scale streaming, peer-to-peer file sharing, or non-compliant VPNs) are operating over that spectrum. This can indicate a violation of the license terms or a deviation from the intended use of the spectrum.<\/li>\n\n\n\n<li>Even for protocols that are encrypted at the payload level, such as <strong>TLS<\/strong>, nDPI can extract <strong>unencrypted metadata from the handshake<\/strong>. This includes the <strong>Server Name Indication (SNI)<\/strong>, which indicates the hostname the client intends to connect to, and <strong>certificate details<\/strong> like the Common Name(s), Issuer DN, and Subject DN. This allows the FCC to understand <em>what services are being accessed<\/em> even if the content is encrypted, providing insight into the nature of the traffic utilizing the spectrum.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Analysis of Traffic Patterns and Volume:<\/strong>\n<ul class=\"wp-block-list\">\n<li>nDPI tracks vital flow statistics such as <strong><code>packets_processed<\/code><\/strong> and <strong><code>total_l4_data_len<\/code><\/strong> for each unique flow. By monitoring these metrics, the FCC can ascertain the <strong>volume and intensity of data transmission<\/strong> associated with specific flows or applications.<\/li>\n\n\n\n<li><strong>Unusual spikes in data volume<\/strong> or sustained high-bandwidth flows for certain applications, especially on spectrum bands with limited capacity or specific usage restrictions, could indicate <strong>inefficient or unauthorized spectrum usage<\/strong>. For instance, an unexpected surge of video streaming traffic on a band allocated for low-bandwidth telemetry could signal a regulatory compliance issue.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Endpoint Identification and Attribution:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Each flow recorded by nDPI includes the <strong>source and destination IP addresses (<code>ip_tuple<\/code>)<\/strong> for both IPv4 and IPv6, as well as <strong>source and destination ports (<code>src_port<\/code>, <code>dst_port<\/code>)<\/strong>.<\/li>\n\n\n\n<li>This detailed addressing information allows the FCC to <strong>identify the communicating devices or entities<\/strong> involved in specific traffic flows. If a particular IP address or port is associated with a device or service that is known to violate spectrum regulations (e.g., an unauthorized broadcaster or a device causing interference), nDPI can pinpoint its network activity.<\/li>\n\n\n\n<li>The ability to convert IP tuples to human-readable strings further aids in this identification.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Detection of Unusual or Malformed Traffic:<\/strong>\n<ul class=\"wp-block-list\">\n<li>nDPI logs instances of <strong>&#8220;Malformed TCP packet&#8221;<\/strong> or <strong>&#8220;Malformed UDP packet&#8221;<\/strong>, and can identify <strong>unknown Layer 4 protocols<\/strong>. While these aren&#8217;t direct spectrum violations, such anomalies in network traffic could potentially be symptoms of <strong>misconfigured devices, network attacks, or other issues<\/strong> that might contribute to unintentional interference or inefficient spectrum use.<\/li>\n\n\n\n<li>The <strong><code>is_midstream_flow<\/code><\/strong> flag for TCP packets can highlight sessions where the SYN handshake was not observed, which might be indicative of unusual connection patterns.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Evidence Collection and Reporting:<\/strong>\n<ul class=\"wp-block-list\">\n<li>The detailed information gathered by nDPI on each flow, including its unique <strong><code>flow_id<\/code><\/strong>, timestamps (<code>first_seen<\/code>, <code>last_seen<\/code>), and detected protocol, can be formatted into a human-readable <strong>flow information string<\/strong> using <code>ndpi_get_flow_info<\/code>.<\/li>\n\n\n\n<li>This comprehensive data serves as <strong>strong forensic evidence<\/strong> that the FCC can use in investigations, enforcement actions, and demonstrating non-compliance with spectrum licenses or regulations. It provides a granular view of <em>what<\/em> traffic was transmitted, <em>when<\/em>, and <em>between whom<\/em>, facilitating a robust enforcement process.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>In essence, nDPI acts as a powerful analytical tool for understanding the &#8220;who, what, and when&#8221; of network traffic, providing the FCC with critical insights into how the radio spectrum is being utilized at the application level, thereby supporting effective spectrum enforcement.<\/p>\n\n\n\n<p>Certainly, let&#8217;s continue to explore how nDPI&#8217;s capabilities can be a valuable asset to the FCC in their Spectrum Enforcement initiatives, building on our previous discussion and the provided source material.<\/p>\n\n\n\n<p>nDPI, by providing deep visibility into network traffic, offers several ways to assist the FCC, even though it doesn&#8217;t directly interact with radio frequencies:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Precise Identification of Application Protocols and Services (Layer 7 DPI)<\/strong> nDPI&#8217;s core strength lies in its ability to identify the <strong>application-layer protocol (<code>detected_l7_protocol<\/code>)<\/strong> for each network flow. This includes categorizing the traffic by its <code>master_protocol<\/code>, <code>app_protocol<\/code>, and <code>category<\/code>. This is profoundly helpful for the FCC because:\n<ul class=\"wp-block-list\">\n<li>Spectrum licenses often dictate specific types of services or applications that are permitted to operate within a given frequency band. For instance, a band might be allocated for low-bandwidth telemetry or specific private network communications. If nDPI identifies <strong>high-bandwidth video streaming (e.g., YouTube, Netflix)<\/strong> or <strong>peer-to-peer file sharing (e.g., BitTorrent)<\/strong> applications on a band licensed for other purposes, it could indicate a violation of license terms or unauthorized use.<\/li>\n\n\n\n<li>The <code>ndpi_detection_process_packet<\/code> function performs this deep packet inspection, and once a protocol is definitively detected, the <code>detection_completed<\/code> flag is set. If a protocol cannot be conclusively identified after a certain number of packets (e.g., <code>0xFF<\/code> or <code>0xFE<\/code>), nDPI attempts to make a <code>guessed_protocol<\/code> classification, providing even &#8220;last chance&#8221; insights into the nature of the traffic.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Extraction of Unencrypted TLS Handshake Metadata<\/strong> Even for encrypted traffic, nDPI can extract crucial unencrypted information from the initial TLS handshake, which is vital for the FCC&#8217;s understanding of the services operating over the spectrum.\n<ul class=\"wp-block-list\">\n<li>From a <strong>TLS Client Hello<\/strong>, nDPI can report the <strong>SSL\/TLS version<\/strong> (<code>ssl_version<\/code>), the <strong>Server Name Indication (SNI)<\/strong> (<code>host_server_name<\/code>), and <strong>Advertised ALPNs<\/strong> (Application-Layer Protocol Negotiation values). The SNI, in particular, reveals the hostname the client is trying to connect to, allowing the FCC to understand <em>which online service<\/em> is being accessed. This helps in identifying whether an entity is, for example, attempting to connect to unauthorized or banned services over a regulated spectrum.<\/li>\n\n\n\n<li>From a <strong>TLS Server Hello<\/strong>, if certificate information is available, nDPI can report the <strong>SSL\/TLS version<\/strong>, <strong>Common Name(s)<\/strong> (<code>server_names<\/code>) from the server&#8217;s certificate, the <strong>Issuer Distinguished Name (DN)<\/strong> (<code>issuerDN<\/code>), and the <strong>Subject Distinguished Name (DN)<\/strong> (<code>subjectDN<\/code>). This information helps the FCC identify the actual server and the certificate authority that issued its certificate, providing more context about the entities and services utilizing the spectrum.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Analysis of Traffic Patterns, Volume, and Flow State<\/strong> nDPI tracks detailed statistics for each flow (<code>nDPI_flow_info<\/code> structure), offering insights into how the spectrum is being utilized in terms of intensity and duration:\n<ul class=\"wp-block-list\">\n<li><strong>Traffic Volume and Intensity<\/strong>: Each flow records <code>packets_processed<\/code> and <code>total_l4_data_len<\/code>. By aggregating this data, the FCC can detect <strong>unusual spikes in data volume or sustained high-bandwidth usage<\/strong>. For instance, if a spectrum band is designated for intermittent, low-data-rate communications, and nDPI consistently identifies large volumes of data for video or file transfers, it signals a potential deviation from the permitted use.<\/li>\n\n\n\n<li><strong>Flow State and Anomalies<\/strong>: The <code>nDPI_flow_info<\/code> structure includes flags like <code>is_midstream_flow<\/code> (indicating a TCP flow where the SYN handshake was not observed, which could be unusual), and <code>flow_fin_ack_seen<\/code> (indicating the end of a TCP connection). The detection of <strong>&#8220;Malformed TCP packet&#8221; or &#8220;Malformed UDP packet&#8221;<\/strong> or <strong>unknown Layer 4 protocols<\/strong> by nDPI could point to misconfigured equipment, unapproved proprietary protocols, or even malicious activity that might contribute to inefficient spectrum use or interference.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Endpoint Identification and Attribution<\/strong> Every flow processed by nDPI contains the <strong>source and destination IP addresses<\/strong> (<code>ip_tuple<\/code> for both IPv4 and IPv6) and <strong>source and destination ports<\/strong> (<code>src_port<\/code>, <code>dst_port<\/code>).\n<ul class=\"wp-block-list\">\n<li>The <code>ip_tuple_to_string<\/code> function allows these IP addresses to be converted into human-readable strings. This granular addressing information enables the FCC to <strong>pinpoint the specific devices or entities<\/strong> involved in suspicious network activity. If an IP address is linked to an unauthorized user or a device known to cause interference, nDPI&#8217;s data can provide direct evidence of its network communications. The unique <code>flow_id<\/code> assigned to each flow further aids in tracking specific conversations.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Comprehensive Evidence Collection and Reporting<\/strong> The detailed flow information gathered by nDPI, including the detected protocol, timestamps (<code>first_seen<\/code>, <code>last_seen<\/code>), and unique flow ID, can be formatted into a human-readable flow information string using <code>ndpi_get_flow_info<\/code>.\n<ul class=\"wp-block-list\">\n<li>This comprehensive data serves as <strong>strong forensic evidence<\/strong> for the FCC. It provides a clear, granular record of &#8220;what&#8221; traffic was transmitted, &#8220;when&#8221; it occurred (<code>time_ms<\/code>, <code>first_seen<\/code>, <code>last_seen<\/code>), &#8220;between whom&#8221; (<code>src_addr_str<\/code>, <code>dst_addr_str<\/code>, <code>src_port<\/code>, <code>dst_port<\/code>), and the &#8220;application&#8221; responsible for the traffic. This level of detail is crucial for documenting non-compliance, issuing warnings, or taking enforcement actions against entities misusing the radio spectrum.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p>Code in Review:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/*\n *\n * Copyright (C) 2011-25 - ntop.org\n *\n * nDPI is free software: you can redistribute it and\/or modify\n * it under the terms of the GNU Lesser General Public License as published by\n * the Free Software Foundation, either version 3 of the License, or\n * (at your option) any later version.\n *\n * nDPI is distributed in the hope that it will be useful,\n * but WITHOUT ANY WARRANTY; without even the implied warranty of\n * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n * GNU Lesser General Public License for more details.\n *\n * You should have received a copy of the GNU Lesser General Public License\n * along with nDPI.  If not, see &lt;http:\/\/www.gnu.org\/licenses\/&gt;.\n *\n *\/\n\n#ifndef WIN32\n#include &lt;arpa\/inet.h&gt;\n#include &lt;netinet\/in.h&gt;\n#endif\n#include &lt;errno.h&gt;\n#include &lt;ndpi_api.h&gt;\n#include &lt;ndpi_main.h&gt;\n#include &lt;ndpi_typedefs.h&gt;\n#include &lt;pcap\/pcap.h&gt;\n#include &lt;pthread.h&gt;\n#include &lt;signal.h&gt;\n#include &lt;stdio.h&gt;\n#include &lt;stdlib.h&gt;\n#include &lt;string.h&gt;\n#include &lt;unistd.h&gt;\n\n#ifdef WIN32\n#include &lt;windows.h&gt;\n#endif\n\n\/\/#define VERBOSE 1\n#define MAX_FLOW_ROOTS_PER_THREAD 2048\n#define MAX_IDLE_FLOWS_PER_THREAD 64\n#define TICK_RESOLUTION 1000\n#define MAX_READER_THREADS 4\n#define IDLE_SCAN_PERIOD 10000 \/* msec *\/\n#define MAX_IDLE_TIME 300000 \/* msec *\/\n#define INITIAL_THREAD_HASH 0x03dd018b\n\n#ifndef ETH_P_IP\n#define ETH_P_IP 0x0800\n#endif\n\n#ifndef ETH_P_IPV6\n#define ETH_P_IPV6 0x86DD\n#endif\n\n#ifndef ETH_P_ARP\n#define ETH_P_ARP  0x0806\n#endif\n\nenum nDPI_l3_type {\n  L3_IP, L3_IP6\n};\n\nstruct nDPI_flow_info {\n  uint32_t flow_id;\n  unsigned long long int packets_processed;\n  uint64_t first_seen;\n  uint64_t last_seen;\n  uint64_t hashval;\n\n  enum nDPI_l3_type l3_type;\n  union {\n    struct {\n      uint32_t src;\n      uint32_t pad_00&#91;3];\n      uint32_t dst;\n      uint32_t pad_01&#91;3];\n    } v4;\n    struct {\n      uint64_t src&#91;2];\n      uint64_t dst&#91;2];\n    } v6;\n\n    struct {\n      uint32_t src&#91;4];\n      uint32_t dst&#91;4];\n    } u32;\n  } ip_tuple;\n\n  unsigned long long int total_l4_data_len;\n  uint16_t src_port;\n  uint16_t dst_port;\n\n  uint8_t is_midstream_flow:1;\n  uint8_t flow_fin_ack_seen:1;\n  uint8_t flow_ack_seen:1;\n  uint8_t detection_completed:1;\n  uint8_t tls_client_hello_seen:1;\n  uint8_t tls_server_hello_seen:1;\n  uint8_t flow_info_printed:1;\n  uint8_t reserved_00:1;\n  uint8_t l4_protocol;\n\n  struct ndpi_proto detected_l7_protocol;\n  struct ndpi_proto guessed_protocol;\n\n  struct ndpi_flow_struct * ndpi_flow;\n};\n\nstruct nDPI_workflow {\n  pcap_t * pcap_handle;\n\n  volatile long int error_or_eof;\n\n  unsigned long long int packets_captured;\n  unsigned long long int packets_processed;\n  unsigned long long int total_l4_data_len;\n  unsigned long long int detected_flow_protocols;\n\n  uint64_t last_idle_scan_time;\n  uint64_t last_time;\n\n  void ** ndpi_flows_active;\n  unsigned long long int max_active_flows;\n  unsigned long long int cur_active_flows;\n  unsigned long long int total_active_flows;\n\n  void ** ndpi_flows_idle;\n  unsigned long long int max_idle_flows;\n  unsigned long long int cur_idle_flows;\n  unsigned long long int total_idle_flows;\n\n  struct ndpi_detection_module_struct * ndpi_struct;\n};\n\nstruct nDPI_reader_thread {\n  struct nDPI_workflow * workflow;\n  pthread_t thread_id;\n  uint32_t array_index;\n};\n\nstatic struct nDPI_reader_thread reader_threads&#91;MAX_READER_THREADS] = {};\nstatic int reader_thread_count = MAX_READER_THREADS;\nstatic volatile long int main_thread_shutdown = 0;\nstatic volatile long int flow_id = 0;\n\nstatic void free_workflow(struct nDPI_workflow ** const workflow);\n\nstatic struct nDPI_workflow * init_workflow(char const * const file_or_device)\n{\n  char pcap_error_buffer&#91;PCAP_ERRBUF_SIZE];\n  struct nDPI_workflow * workflow = (struct nDPI_workflow *)ndpi_calloc(1, sizeof(*workflow));\n  const char *bpfFilter = \"ip or ip6\";\n  static struct bpf_program bpf_code;\n  static struct bpf_program *bpf_cfilter = NULL;\n  \n  if (workflow == NULL) {\n    return NULL;\n  }\n\n  if (access(file_or_device, R_OK) != 0 &amp;&amp; errno == ENOENT) {\n    workflow-&gt;pcap_handle = pcap_open_live(file_or_device, \/* 1536 *\/ 65535, 1, 250, pcap_error_buffer);\n  } else {\n#ifdef WIN32\n    workflow-&gt;pcap_handle = pcap_open_offline(file_or_device, pcap_error_buffer);\n#else\n    workflow-&gt;pcap_handle = pcap_open_offline_with_tstamp_precision(file_or_device, PCAP_TSTAMP_PRECISION_MICRO,\n\t\t\t\t\t\t\t\t    pcap_error_buffer);\n#endif\n  }\n\n  if (workflow-&gt;pcap_handle == NULL) {\n    fprintf(stderr, \"pcap_open_live \/ pcap_open_offline: %.*s\\n\",\n\t    (int) PCAP_ERRBUF_SIZE, pcap_error_buffer);\n    free_workflow(&amp;workflow);\n    return NULL;\n  }\n\n  if(pcap_compile(workflow-&gt;pcap_handle, &amp;bpf_code, bpfFilter, 1, 0xFFFFFF00) &lt; 0) {\n    printf(\"pcap_compile error: '%s'\\n\", pcap_geterr(workflow-&gt;pcap_handle));\n    exit(-1);\n  }\n\n  bpf_cfilter = &amp;bpf_code;\n\n  if(pcap_setfilter(workflow-&gt;pcap_handle, bpf_cfilter) &lt; 0) {\n    printf(\"pcap_setfilter error: '%s'\\n\", pcap_geterr(workflow-&gt;pcap_handle));\n  }  \n\n  workflow-&gt;ndpi_struct = ndpi_init_detection_module(NULL);\n  if (workflow-&gt;ndpi_struct == NULL) {\n    free_workflow(&amp;workflow);\n    return NULL;\n  }\n\n  workflow-&gt;total_active_flows = 0;\n  workflow-&gt;max_active_flows = MAX_FLOW_ROOTS_PER_THREAD;\n  workflow-&gt;ndpi_flows_active = (void **)ndpi_calloc(workflow-&gt;max_active_flows, sizeof(void *));\n  if (workflow-&gt;ndpi_flows_active == NULL) {\n    free_workflow(&amp;workflow);\n    return NULL;\n  }\n\n  workflow-&gt;total_idle_flows = 0;\n  workflow-&gt;max_idle_flows = MAX_IDLE_FLOWS_PER_THREAD;\n  workflow-&gt;ndpi_flows_idle = (void **)ndpi_calloc(workflow-&gt;max_idle_flows, sizeof(void *));\n  if (workflow-&gt;ndpi_flows_idle == NULL) {\n    free_workflow(&amp;workflow);\n    return NULL;\n  }\n\n  NDPI_PROTOCOL_BITMASK protos;\n  NDPI_BITMASK_SET_ALL(protos);\n  ndpi_set_protocol_detection_bitmask2(workflow-&gt;ndpi_struct, &amp;protos);\n  ndpi_finalize_initialization(workflow-&gt;ndpi_struct);\n\n  return workflow;\n}\n\nstatic void ndpi_flow_info_freer(void * const node)\n{\n  struct nDPI_flow_info * const flow = (struct nDPI_flow_info *)node;\n\n  ndpi_flow_free(flow-&gt;ndpi_flow);\n  ndpi_free(flow);\n}\n\nstatic void free_workflow(struct nDPI_workflow ** const workflow)\n{\n  struct nDPI_workflow * const w = *workflow;\n  size_t i;\n\n  if (w == NULL) {\n    return;\n  }\n\n  if (w-&gt;pcap_handle != NULL) {\n    pcap_close(w-&gt;pcap_handle);\n    w-&gt;pcap_handle = NULL;\n  }\n\n  if (w-&gt;ndpi_struct != NULL) {\n    ndpi_exit_detection_module(w-&gt;ndpi_struct);\n  }\n  for(i = 0; i &lt; w-&gt;max_active_flows; i++) {\n    ndpi_tdestroy(w-&gt;ndpi_flows_active&#91;i], ndpi_flow_info_freer);\n  }\n  ndpi_free(w-&gt;ndpi_flows_active);\n  ndpi_free(w-&gt;ndpi_flows_idle);\n  ndpi_free(w);\n  *workflow = NULL;\n}\n\nstatic char * get_default_pcapdev(char *errbuf)\n{\n  char * ifname;\n  pcap_if_t * all_devices = NULL;\n\n  if (pcap_findalldevs(&amp;all_devices, errbuf) != 0)\n    {\n      return NULL;\n    }\n\n  ifname = strdup(all_devices&#91;0].name);\n  pcap_freealldevs(all_devices);\n\n  return ifname;\n}\n\nstatic int setup_reader_threads(char const * const file_or_device)\n{\n  char * file_or_default_device;\n  char pcap_error_buffer&#91;PCAP_ERRBUF_SIZE];\n  int i;\n\n  if (reader_thread_count &gt; MAX_READER_THREADS) {\n    return 1;\n  }\n\n  if (file_or_device == NULL) {\n    file_or_default_device = get_default_pcapdev(pcap_error_buffer);\n    if (file_or_default_device == NULL) {\n      fprintf(stderr, \"pcap_findalldevs: %.*s\\n\", (int) PCAP_ERRBUF_SIZE, pcap_error_buffer);\n      return 1;\n    }\n  } else {\n    file_or_default_device = strdup(file_or_device);\n    if (file_or_default_device == NULL) {\n      return 1;\n    }\n  }\n\n  for (i = 0; i &lt; reader_thread_count; ++i) {\n    reader_threads&#91;i].workflow = init_workflow(file_or_default_device);\n    if (reader_threads&#91;i].workflow == NULL)\n      {\n\tfree(file_or_default_device);\n\treturn 1;\n      }\n  }\n\n  free(file_or_default_device);\n  return 0;\n}\n\nstatic int ip_tuple_to_string(struct nDPI_flow_info const * const flow,\n                              char * const src_addr_str, size_t src_addr_len,\n                              char * const dst_addr_str, size_t dst_addr_len)\n{\n  switch (flow-&gt;l3_type) {\n  case L3_IP:\n    return inet_ntop(AF_INET, (struct sockaddr_in *)&amp;flow-&gt;ip_tuple.v4.src,\n\t\t     src_addr_str, src_addr_len) != NULL &amp;&amp;\n      inet_ntop(AF_INET, (struct sockaddr_in *)&amp;flow-&gt;ip_tuple.v4.dst,\n\t\tdst_addr_str, dst_addr_len) != NULL;\n  case L3_IP6:\n    return inet_ntop(AF_INET6, (struct sockaddr_in6 *)&amp;flow-&gt;ip_tuple.v6.src&#91;0],\n\t\t     src_addr_str, src_addr_len) != NULL &amp;&amp;\n      inet_ntop(AF_INET6, (struct sockaddr_in6 *)&amp;flow-&gt;ip_tuple.v6.dst&#91;0],\n\t\tdst_addr_str, dst_addr_len) != NULL;\n  }\n\n  return 0;\n}\n\n#ifdef VERBOSE\nstatic void print_packet_info(struct nDPI_reader_thread const * const reader_thread,\n                              struct pcap_pkthdr const * const header,\n                              uint32_t l4_data_len,\n                              struct nDPI_flow_info const * const flow)\n{\n  struct nDPI_workflow const * const workflow = reader_thread-&gt;workflow;\n  char src_addr_str&#91;INET6_ADDRSTRLEN+1] = {0};\n  char dst_addr_str&#91;INET6_ADDRSTRLEN+1] = {0};\n  char buf&#91;256];\n  int used = 0, ret;\n\n  ret = ndpi_snprintf(buf, sizeof(buf), \"&#91;%8llu, %d, %4u] %4u bytes: \",\n\t\t workflow-&gt;packets_captured, reader_thread-&gt;array_index,\n\t\t flow-&gt;flow_id, header-&gt;caplen);\n  if (ret &gt; 0) {\n    used += ret;\n  }\n\n  if (ip_tuple_to_string(flow, src_addr_str, sizeof(src_addr_str), dst_addr_str, sizeof(dst_addr_str)) != 0) {\n    ret = ndpi_snprintf(buf + used, sizeof(buf) - used, \"IP&#91;%s -&gt; %s]\", src_addr_str, dst_addr_str);\n  } else {\n    ret = ndpi_snprintf(buf + used, sizeof(buf) - used, \"IP&#91;ERROR]\");\n  }\n  if (ret &gt; 0) {\n    used += ret;\n  }\n\n  switch (flow-&gt;l4_protocol) {\n  case IPPROTO_UDP:\n    ret = ndpi_snprintf(buf + used, sizeof(buf) - used, \" -&gt; UDP&#91;%u -&gt; %u, %u bytes]\",\n\t\t   flow-&gt;src_port, flow-&gt;dst_port, l4_data_len);\n    break;\n  case IPPROTO_TCP:\n    ret = ndpi_snprintf(buf + used, sizeof(buf) - used, \" -&gt; TCP&#91;%u -&gt; %u, %u bytes]\",\n\t\t   flow-&gt;src_port, flow-&gt;dst_port, l4_data_len);\n    break;\n  case IPPROTO_ICMP:\n    ret = ndpi_snprintf(buf + used, sizeof(buf) - used, \" -&gt; ICMP\");\n    break;\n  case IPPROTO_ICMPV6:\n    ret = ndpi_snprintf(buf + used, sizeof(buf) - used, \" -&gt; ICMP6\");\n    break;\n  case IPPROTO_HOPOPTS:\n    ret = ndpi_snprintf(buf + used, sizeof(buf) - used, \" -&gt; ICMP6 Hop-By-Hop\");\n    break;\n  default:\n    ret = ndpi_snprintf(buf + used, sizeof(buf) - used, \" -&gt; Unknown&#91;0x%X]\", flow-&gt;l4_protocol);\n    break;\n  }\n  if (ret &gt; 0) {\n    used += ret;\n  }\n\n  printf(\"%.*s\\n\", used, buf);\n}\n#endif\n\nstatic int ip_tuples_compare(struct nDPI_flow_info const * const A, struct nDPI_flow_info const * const B)\n{\n  \/\/ generate a warning if the enum changes\n  switch (A-&gt;l3_type)\n  {\n    case L3_IP:\n    case L3_IP6:\n      break;\n  }\n\n  if (A-&gt;l3_type == L3_IP &amp;&amp; B-&gt;l3_type == L3_IP)\n  {\n    if (A-&gt;ip_tuple.v4.src &lt; B-&gt;ip_tuple.v4.src)\n    {\n      return -1;\n    }\n    if (A-&gt;ip_tuple.v4.src &gt; B-&gt;ip_tuple.v4.src)\n    {\n      return 1;\n    }\n    if (A-&gt;ip_tuple.v4.dst &lt; B-&gt;ip_tuple.v4.dst)\n    {\n      return -1;\n    }\n    if (A-&gt;ip_tuple.v4.dst &gt; B-&gt;ip_tuple.v4.dst)\n    {\n      return 1;\n    }\n  }\n  else if (A-&gt;l3_type == L3_IP6 &amp;&amp; B-&gt;l3_type == L3_IP6)\n  {\n    if (A-&gt;ip_tuple.v6.src&#91;0] &lt; B-&gt;ip_tuple.v6.src&#91;0] &amp;&amp; A-&gt;ip_tuple.v6.src&#91;1] &lt; B-&gt;ip_tuple.v6.src&#91;1])\n    {\n      return -1;\n    }\n    if (A-&gt;ip_tuple.v6.src&#91;0] &gt; B-&gt;ip_tuple.v6.src&#91;0] &amp;&amp; A-&gt;ip_tuple.v6.src&#91;1] &gt; B-&gt;ip_tuple.v6.src&#91;1])\n    {\n      return 1;\n    }\n    if (A-&gt;ip_tuple.v6.dst&#91;0] &lt; B-&gt;ip_tuple.v6.dst&#91;0] &amp;&amp; A-&gt;ip_tuple.v6.dst&#91;1] &lt; B-&gt;ip_tuple.v6.dst&#91;1])\n    {\n      return -1;\n    }\n    if (A-&gt;ip_tuple.v6.dst&#91;0] &gt; B-&gt;ip_tuple.v6.dst&#91;0] &amp;&amp; A-&gt;ip_tuple.v6.dst&#91;1] &gt; B-&gt;ip_tuple.v6.dst&#91;1])\n    {\n      return 1;\n    }\n  }\n\n  if (A-&gt;src_port &lt; B-&gt;src_port)\n  {\n    return -1;\n  }\n  if (A-&gt;src_port &gt; B-&gt;src_port)\n  {\n    return 1;\n  }\n  if (A-&gt;dst_port &lt; B-&gt;dst_port)\n  {\n    return -1;\n  }\n  if (A-&gt;dst_port &gt; B-&gt;dst_port)\n  {\n    return 1;\n  }\n\n  return 0;\n}\n\nstatic void ndpi_idle_scan_walker(void const * const A, ndpi_VISIT which, int depth, void * const user_data)\n{\n  struct nDPI_workflow * const workflow = (struct nDPI_workflow *)user_data;\n  struct nDPI_flow_info * const flow = *(struct nDPI_flow_info **)A;\n\n  (void)depth;\n\n  if (workflow == NULL || flow == NULL) {\n    return;\n  }\n\n  if (workflow-&gt;cur_idle_flows == MAX_IDLE_FLOWS_PER_THREAD) {\n    return;\n  }\n\n  if (which == ndpi_preorder || which == ndpi_leaf) {\n    if ((flow-&gt;flow_fin_ack_seen == 1 &amp;&amp; flow-&gt;flow_ack_seen == 1) ||\n\tflow-&gt;last_seen + MAX_IDLE_TIME &lt; workflow-&gt;last_time)\n      {\n\tchar src_addr_str&#91;INET6_ADDRSTRLEN+1];\n\tchar dst_addr_str&#91;INET6_ADDRSTRLEN+1];\n\tip_tuple_to_string(flow, src_addr_str, sizeof(src_addr_str), dst_addr_str, sizeof(dst_addr_str));\n\tworkflow-&gt;ndpi_flows_idle&#91;workflow-&gt;cur_idle_flows++] = flow;\n\tworkflow-&gt;total_idle_flows++;\n      }\n  }\n}\n\nstatic int ndpi_workflow_node_cmp(void const * const A, void const * const B) {\n  struct nDPI_flow_info const * const flow_info_a = (struct nDPI_flow_info *)A;\n  struct nDPI_flow_info const * const flow_info_b = (struct nDPI_flow_info *)B;\n\n  if (flow_info_a-&gt;hashval &lt; flow_info_b-&gt;hashval) {\n    return(-1);\n  } else if (flow_info_a-&gt;hashval &gt; flow_info_b-&gt;hashval) {\n    return(1);\n  }\n\n  \/* Flows have the same hash *\/\n  if (flow_info_a-&gt;l4_protocol &lt; flow_info_b-&gt;l4_protocol) {\n    return(-1);\n  } else if (flow_info_a-&gt;l4_protocol &gt; flow_info_b-&gt;l4_protocol) {\n    return(1);\n  }\n\n  return ip_tuples_compare(flow_info_a, flow_info_b);\n}\n\nstatic void check_for_idle_flows(struct nDPI_workflow * const workflow)\n{\n  size_t idle_scan_index;\n\n  if (workflow-&gt;last_idle_scan_time + IDLE_SCAN_PERIOD &lt; workflow-&gt;last_time) {\n    for (idle_scan_index = 0; idle_scan_index &lt; workflow-&gt;max_active_flows; ++idle_scan_index) {\n      ndpi_twalk(workflow-&gt;ndpi_flows_active&#91;idle_scan_index], ndpi_idle_scan_walker, workflow);\n\n      while (workflow-&gt;cur_idle_flows &gt; 0) {\n\tstruct nDPI_flow_info * const f =\n\t  (struct nDPI_flow_info *)workflow-&gt;ndpi_flows_idle&#91;--workflow-&gt;cur_idle_flows];\n\tif (f-&gt;flow_fin_ack_seen == 1) {\n\t  printf(\"Free fin flow with id %u\\n\", f-&gt;flow_id);\n\t} else {\n\t  printf(\"Free idle flow with id %u\\n\", f-&gt;flow_id);\n\t}\n\tndpi_tdelete(f, &amp;workflow-&gt;ndpi_flows_active&#91;idle_scan_index],\n\t\t     ndpi_workflow_node_cmp);\n\tndpi_flow_info_freer(f);\n\tworkflow-&gt;cur_active_flows--;\n      }\n    }\n\n    workflow-&gt;last_idle_scan_time = workflow-&gt;last_time;\n  }\n}\n\nstatic void ndpi_process_packet(uint8_t * const args,\n                                struct pcap_pkthdr const * const header,\n                                uint8_t const * const packet)\n{\n  struct nDPI_reader_thread * const reader_thread =\n    (struct nDPI_reader_thread *)args;\n  struct nDPI_workflow * workflow;\n  struct nDPI_flow_info flow;\n\n  size_t hashed_index;\n  void * tree_result;\n  struct nDPI_flow_info * flow_to_process;\n\n  const struct ndpi_ethhdr * ethernet;\n  const struct ndpi_iphdr * ip;\n  struct ndpi_ipv6hdr * ip6;\n\n  uint64_t time_ms;\n  const uint16_t eth_offset = 0;\n  uint16_t ip_offset;\n  uint16_t ip_size;\n\n  const uint8_t * l4_ptr = NULL;\n  uint16_t l4_len = 0;\n\n  uint16_t type;\n  uint32_t thread_index = INITIAL_THREAD_HASH; \/\/ generated with `dd if=\/dev\/random bs=1024 count=1 |&amp; hd'\n\n  memset(&amp;flow, '\\0', sizeof(flow));\n\n  if (reader_thread == NULL) {\n    return;\n  }\n  workflow = reader_thread-&gt;workflow;\n\n  if (workflow == NULL) {\n    return;\n  }\n\n  workflow-&gt;packets_captured++;\n  time_ms = ((uint64_t) header-&gt;ts.tv_sec) * TICK_RESOLUTION + header-&gt;ts.tv_usec \/ (1000000 \/ TICK_RESOLUTION);\n  workflow-&gt;last_time = time_ms;\n\n  check_for_idle_flows(workflow);\n\n  \/* process datalink layer *\/\n  switch (pcap_datalink(workflow-&gt;pcap_handle)) {\n  case DLT_NULL:\n    if (ntohl(*((uint32_t *)&amp;packet&#91;eth_offset])) == 0x00000002) {\n      type = ETH_P_IP;\n    } else {\n      type = ETH_P_IPV6;\n    }\n    ip_offset = 4 + eth_offset;\n    break;\n  case DLT_EN10MB:\n    if (header-&gt;len &lt; sizeof(struct ndpi_ethhdr)) {\n      fprintf(stderr, \"&#91;%8llu, %d] Ethernet packet too short - skipping\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index);\n      return;\n    }\n    ethernet = (struct ndpi_ethhdr *) &amp;packet&#91;eth_offset];\n    ip_offset = sizeof(struct ndpi_ethhdr) + eth_offset;\n    type = ntohs(ethernet-&gt;h_proto);\n    switch (type) {\n    case ETH_P_IP: \/* IPv4 *\/\n      if (header-&gt;len &lt; sizeof(struct ndpi_ethhdr) + sizeof(struct ndpi_iphdr)) {\n\tfprintf(stderr, \"&#91;%8llu, %d] IP packet too short - skipping\\n\",\n\t\tworkflow-&gt;packets_captured, reader_thread-&gt;array_index);\n\treturn;\n      }\n      break;\n    case ETH_P_IPV6: \/* IPV6 *\/\n      if (header-&gt;len &lt; sizeof(struct ndpi_ethhdr) + sizeof(struct ndpi_ipv6hdr)) {\n\tfprintf(stderr, \"&#91;%8llu, %d] IP6 packet too short - skipping\\n\",\n\t\tworkflow-&gt;packets_captured, reader_thread-&gt;array_index);\n\treturn;\n      }\n      break;\n    case ETH_P_ARP: \/* ARP *\/\n      return;\n    default:\n      fprintf(stderr, \"&#91;%8llu, %d] Unknown Ethernet packet with type 0x%X - skipping\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index, type);\n      return;\n    }\n    break;\n  default:\n    fprintf(stderr, \"&#91;%8llu, %d] Captured non IP\/Ethernet packet with datalink type 0x%X - skipping\\n\",\n\t    workflow-&gt;packets_captured, reader_thread-&gt;array_index, pcap_datalink(workflow-&gt;pcap_handle));\n    return;\n  }\n\n  if (type == ETH_P_IP) {\n    ip = (struct ndpi_iphdr *)&amp;packet&#91;ip_offset];\n    ip6 = NULL;\n  } else if (type == ETH_P_IPV6) {\n    ip = NULL;\n    ip6 = (struct ndpi_ipv6hdr *)&amp;packet&#91;ip_offset];\n  } else {\n    fprintf(stderr, \"&#91;%8llu, %d] Captured non IPv4\/IPv6 packet with type 0x%X - skipping\\n\",\n\t    workflow-&gt;packets_captured, reader_thread-&gt;array_index, type);\n    return;\n  }\n  ip_size = header-&gt;len - ip_offset;\n\n  if (type == ETH_P_IP &amp;&amp; header-&gt;len &gt;= ip_offset) {\n    if (header-&gt;caplen &lt; header-&gt;len) {\n      fprintf(stderr, \"&#91;%8llu, %d] Captured packet size is smaller than packet size: %u &lt; %u\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index, header-&gt;caplen, header-&gt;len);\n    }\n  }\n\n  \/* process layer3 e.g. IPv4 \/ IPv6 *\/\n  if (ip != NULL &amp;&amp; ip-&gt;version == 4) {\n    if (ip_size &lt; sizeof(*ip)) {\n      fprintf(stderr, \"&#91;%8llu, %d] Packet smaller than IP4 header length: %u &lt; %zu\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index, ip_size, sizeof(*ip));\n      return;\n    }\n\n    flow.l3_type = L3_IP;\n    if (ndpi_detection_get_l4((uint8_t*)ip, ip_size, &amp;l4_ptr, &amp;l4_len,\n\t\t\t      &amp;flow.l4_protocol, NDPI_DETECTION_ONLY_IPV4) != 0)\n      {\n\tfprintf(stderr, \"&#91;%8llu, %d] nDPI IPv4\/L4 payload detection failed, L4 length: %zu\\n\",\n\t\tworkflow-&gt;packets_captured, reader_thread-&gt;array_index, ip_size - sizeof(*ip));\n\treturn;\n      }\n\n    flow.ip_tuple.v4.src = ip-&gt;saddr;\n    flow.ip_tuple.v4.dst = ip-&gt;daddr;\n    uint32_t min_addr = (flow.ip_tuple.v4.src &gt; flow.ip_tuple.v4.dst ?\n\t\t\t flow.ip_tuple.v4.dst : flow.ip_tuple.v4.src);\n    thread_index = min_addr + ip-&gt;protocol;\n  } else if (ip6 != NULL) {\n    if (ip_size &lt; sizeof(ip6-&gt;ip6_hdr)) {\n      fprintf(stderr, \"&#91;%8llu, %d] Packet smaller than IP6 header length: %u &lt; %zu\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index, ip_size, sizeof(ip6-&gt;ip6_hdr));\n      return;\n    }\n\n    flow.l3_type = L3_IP6;\n    if (ndpi_detection_get_l4((uint8_t*)ip6, ip_size, &amp;l4_ptr, &amp;l4_len,\n\t\t\t      &amp;flow.l4_protocol, NDPI_DETECTION_ONLY_IPV6) != 0)\n      {\n\tfprintf(stderr, \"&#91;%8llu, %d] nDPI IPv6\/L4 payload detection failed, L4 length: %zu\\n\",\n\t\tworkflow-&gt;packets_captured, reader_thread-&gt;array_index, ip_size - sizeof(*ip6));\n\treturn;\n      }\n\n    flow.ip_tuple.v6.src&#91;0] = ip6-&gt;ip6_src.u6_addr.u6_addr64&#91;0];\n    flow.ip_tuple.v6.src&#91;1] = ip6-&gt;ip6_src.u6_addr.u6_addr64&#91;1];\n    flow.ip_tuple.v6.dst&#91;0] = ip6-&gt;ip6_dst.u6_addr.u6_addr64&#91;0];\n    flow.ip_tuple.v6.dst&#91;1] = ip6-&gt;ip6_dst.u6_addr.u6_addr64&#91;1];\n    uint64_t min_addr&#91;2];\n    if (flow.ip_tuple.v6.src&#91;0] &gt; flow.ip_tuple.v6.dst&#91;0] &amp;&amp;\n\tflow.ip_tuple.v6.src&#91;1] &gt; flow.ip_tuple.v6.dst&#91;1])\n      {\n\tmin_addr&#91;0] = flow.ip_tuple.v6.dst&#91;0];\n\tmin_addr&#91;1] = flow.ip_tuple.v6.dst&#91;0];\n      } else {\n      min_addr&#91;0] = flow.ip_tuple.v6.src&#91;0];\n      min_addr&#91;1] = flow.ip_tuple.v6.src&#91;0];\n    }\n    thread_index = min_addr&#91;0] + min_addr&#91;1] + ip6-&gt;ip6_hdr.ip6_un1_nxt;\n  } else {\n    fprintf(stderr, \"&#91;%8llu, %d] Non IP\/IPv6 protocol detected: 0x%X\\n\",\n\t    workflow-&gt;packets_captured, reader_thread-&gt;array_index, type);\n    return;\n  }\n\n  \/* process layer4 e.g. TCP \/ UDP *\/\n  if (flow.l4_protocol == IPPROTO_TCP) {\n    const struct ndpi_tcphdr * tcp;\n\n    if (header-&gt;len &lt; (l4_ptr - packet) + sizeof(struct ndpi_tcphdr)) {\n      fprintf(stderr, \"&#91;%8llu, %d] Malformed TCP packet, packet size smaller than expected: %u &lt; %zu\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index,\n\t      header-&gt;len, (l4_ptr - packet) + sizeof(struct ndpi_tcphdr));\n      return;\n    }\n    tcp = (struct ndpi_tcphdr *)l4_ptr;\n    flow.is_midstream_flow = (tcp-&gt;syn == 0 ? 1 : 0);\n    flow.flow_fin_ack_seen = (tcp-&gt;fin == 1 &amp;&amp; tcp-&gt;ack == 1 ? 1 : 0);\n    flow.flow_ack_seen = tcp-&gt;ack;\n    flow.src_port = ntohs(tcp-&gt;source);\n    flow.dst_port = ntohs(tcp-&gt;dest);\n  } else if (flow.l4_protocol == IPPROTO_UDP) {\n    const struct ndpi_udphdr * udp;\n\n    if (header-&gt;len &lt; (l4_ptr - packet) + sizeof(struct ndpi_udphdr)) {\n      fprintf(stderr, \"&#91;%8llu, %d] Malformed UDP packet, packet size smaller than expected: %u &lt; %zu\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index,\n\t      header-&gt;len, (l4_ptr - packet) + sizeof(struct ndpi_udphdr));\n      return;\n    }\n    udp = (struct ndpi_udphdr *)l4_ptr;\n    flow.src_port = ntohs(udp-&gt;source);\n    flow.dst_port = ntohs(udp-&gt;dest);\n  }\n\n  \/* distribute flows to threads while keeping stability (same flow goes always to same thread) *\/\n  thread_index += (flow.src_port &lt; flow.dst_port ? flow.dst_port : flow.src_port);\n  thread_index %= reader_thread_count;\n  if (thread_index != reader_thread-&gt;array_index) {\n    return;\n  }\n  workflow-&gt;packets_processed++;\n  workflow-&gt;total_l4_data_len += l4_len;\n\n#ifdef VERBOSE\n  print_packet_info(reader_thread, header, l4_len, &amp;flow);\n#endif\n\n  {\n    uint64_t tmp&#91;4] = {};\n\n    \/* calculate flow hash for btree find, search(insert) *\/\n    if (flow.l3_type == L3_IP) {\n      if (ndpi_flowv4_flow_hash(flow.l4_protocol, flow.ip_tuple.v4.src, flow.ip_tuple.v4.dst,\n          flow.src_port, flow.dst_port, 0, 0,\n          (uint8_t *)&amp;tmp&#91;0], sizeof(tmp)) != 0)\n      {\n        flow.hashval = flow.ip_tuple.v4.src + flow.ip_tuple.v4.dst; \/\/ fallback\n      } else {\n        flow.hashval = tmp&#91;0] + tmp&#91;1] + tmp&#91;2] + tmp&#91;3];\n      }\n    } else if (flow.l3_type == L3_IP6) {\n      if (ndpi_flowv6_flow_hash(flow.l4_protocol, &amp;ip6-&gt;ip6_src, &amp;ip6-&gt;ip6_dst,\n          flow.src_port, flow.dst_port, 0, 0,\n          (uint8_t *)&amp;tmp&#91;0], sizeof(tmp)) != 0)\n      {\n        flow.hashval = flow.ip_tuple.v6.src&#91;0] + flow.ip_tuple.v6.src&#91;1];\n        flow.hashval += flow.ip_tuple.v6.dst&#91;0] + flow.ip_tuple.v6.dst&#91;1];\n      } else {\n        flow.hashval = tmp&#91;0] + tmp&#91;1] + tmp&#91;2] + tmp&#91;3];\n      }\n    }\n\n    flow.hashval += flow.l4_protocol + flow.src_port + flow.dst_port;\n  }\n\n  hashed_index = flow.hashval % workflow-&gt;max_active_flows;\n  tree_result = ndpi_tfind(&amp;flow, &amp;workflow-&gt;ndpi_flows_active&#91;hashed_index], ndpi_workflow_node_cmp);\n  if (tree_result == NULL) {\n    \/* flow not found in btree: switch src &lt;-&gt; dst and try to find it again *\/\n    uint32_t orig_src_ip&#91;4] = { flow.ip_tuple.u32.src&#91;0], flow.ip_tuple.u32.src&#91;1],\n                                flow.ip_tuple.u32.src&#91;2], flow.ip_tuple.u32.src&#91;3] };\n    uint32_t orig_dst_ip&#91;4] = { flow.ip_tuple.u32.dst&#91;0], flow.ip_tuple.u32.dst&#91;1],\n                                flow.ip_tuple.u32.dst&#91;2], flow.ip_tuple.u32.dst&#91;3] };\n    uint16_t orig_src_port = flow.src_port;\n    uint16_t orig_dst_port = flow.dst_port;\n\n    flow.ip_tuple.u32.src&#91;0] = orig_dst_ip&#91;0];\n    flow.ip_tuple.u32.src&#91;1] = orig_dst_ip&#91;1];\n    flow.ip_tuple.u32.src&#91;2] = orig_dst_ip&#91;2];\n    flow.ip_tuple.u32.src&#91;3] = orig_dst_ip&#91;3];\n\n    flow.ip_tuple.u32.dst&#91;0] = orig_src_ip&#91;0];\n    flow.ip_tuple.u32.dst&#91;1] = orig_src_ip&#91;1];\n    flow.ip_tuple.u32.dst&#91;2] = orig_src_ip&#91;2];\n    flow.ip_tuple.u32.dst&#91;3] = orig_src_ip&#91;3];\n\n    flow.src_port = orig_dst_port;\n    flow.dst_port = orig_src_port;\n\n    tree_result = ndpi_tfind(&amp;flow, &amp;workflow-&gt;ndpi_flows_active&#91;hashed_index], ndpi_workflow_node_cmp);\n\n    flow.ip_tuple.u32.src&#91;0] = orig_src_ip&#91;0];\n    flow.ip_tuple.u32.src&#91;1] = orig_src_ip&#91;1];\n    flow.ip_tuple.u32.src&#91;2] = orig_src_ip&#91;2];\n    flow.ip_tuple.u32.src&#91;3] = orig_src_ip&#91;3];\n\n    flow.ip_tuple.u32.dst&#91;0] = orig_dst_ip&#91;0];\n    flow.ip_tuple.u32.dst&#91;1] = orig_dst_ip&#91;1];\n    flow.ip_tuple.u32.dst&#91;2] = orig_dst_ip&#91;2];\n    flow.ip_tuple.u32.dst&#91;3] = orig_dst_ip&#91;3];\n\n    flow.src_port = orig_src_port;\n    flow.dst_port = orig_dst_port;\n  }\n\n  if (tree_result == NULL) {\n    \/* flow still not found, must be new *\/\n    if (workflow-&gt;cur_active_flows == workflow-&gt;max_active_flows) {\n      fprintf(stderr, \"&#91;%8llu, %d] max flows to track reached: %llu, idle: %llu\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index,\n\t      workflow-&gt;max_active_flows, workflow-&gt;cur_idle_flows);\n      return;\n    }\n\n    flow_to_process = (struct nDPI_flow_info *)ndpi_malloc(sizeof(*flow_to_process));\n    if (flow_to_process == NULL) {\n      fprintf(stderr, \"&#91;%8llu, %d] Not enough memory for flow info\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index);\n      return;\n    }\n\n    memcpy(flow_to_process, &amp;flow, sizeof(*flow_to_process));\n    flow_to_process-&gt;flow_id = __sync_fetch_and_add(&amp;flow_id, 1);\n\n    flow_to_process-&gt;ndpi_flow = (struct ndpi_flow_struct *)ndpi_flow_malloc(SIZEOF_FLOW_STRUCT);\n    if (flow_to_process-&gt;ndpi_flow == NULL) {\n      fprintf(stderr, \"&#91;%8llu, %d, %4u] Not enough memory for flow struct\\n\",\n\t      workflow-&gt;packets_captured, reader_thread-&gt;array_index, flow_to_process-&gt;flow_id);\n      return;\n    }\n    memset(flow_to_process-&gt;ndpi_flow, 0, SIZEOF_FLOW_STRUCT);\n\n    printf(\"&#91;%8llu, %d, %4u] new %sflow\\n\", workflow-&gt;packets_captured, thread_index,\n\t   flow_to_process-&gt;flow_id,\n\t   (flow_to_process-&gt;is_midstream_flow != 0 ? \"midstream-\" : \"\"));\n    if (ndpi_tsearch(flow_to_process, &amp;workflow-&gt;ndpi_flows_active&#91;hashed_index], ndpi_workflow_node_cmp) == NULL) {\n      \/* Possible Leak, but should not happen as we'd abort earlier. *\/\n      return;\n    }\n\n    workflow-&gt;cur_active_flows++;\n    workflow-&gt;total_active_flows++;\n  } else {\n    flow_to_process = *(struct nDPI_flow_info **)tree_result;\n  }\n\n  flow_to_process-&gt;packets_processed++;\n  flow_to_process-&gt;total_l4_data_len += l4_len;\n  \/* update timestamps, important for timeout handling *\/\n  if (flow_to_process-&gt;first_seen == 0) {\n    flow_to_process-&gt;first_seen = time_ms;\n  }\n  flow_to_process-&gt;last_seen = time_ms;\n  \/* current packet is an TCP-ACK? *\/\n  flow_to_process-&gt;flow_ack_seen = flow.flow_ack_seen;\n\n  \/* TCP-FIN: indicates that at least one side wants to end the connection *\/\n  if (flow.flow_fin_ack_seen != 0 &amp;&amp; flow_to_process-&gt;flow_fin_ack_seen == 0) {\n    flow_to_process-&gt;flow_fin_ack_seen = 1;\n    printf(\"&#91;%8llu, %d, %4u] end of flow\\n\",  workflow-&gt;packets_captured, thread_index,\n\t   flow_to_process-&gt;flow_id);\n    return;\n  }\n\n  \/*\n   * This example tries to use maximum supported packets for detection:\n   * for uint8: 0xFF\n   *\/\n  if (flow_to_process-&gt;ndpi_flow-&gt;num_processed_pkts == 0xFF) {\n    return;\n  } else if (flow_to_process-&gt;ndpi_flow-&gt;num_processed_pkts == 0xFE) {\n    \/* last chance to guess something, better then nothing *\/\n    uint8_t protocol_was_guessed = 0;\n    flow_to_process-&gt;guessed_protocol =\n      ndpi_detection_giveup(workflow-&gt;ndpi_struct,\n\t\t\t    flow_to_process-&gt;ndpi_flow,\n\t\t\t    &amp;protocol_was_guessed);\n    if (protocol_was_guessed != 0) {\n      printf(\"&#91;%8llu, %d, %4d]&#91;GUESSED] protocol: %s | app protocol: %s | category: %s\\n\",\n\t     workflow-&gt;packets_captured,\n\t     reader_thread-&gt;array_index,\n\t     flow_to_process-&gt;flow_id,\n\t     ndpi_get_proto_name(workflow-&gt;ndpi_struct, flow_to_process-&gt;guessed_protocol.proto.master_protocol),\n\t     ndpi_get_proto_name(workflow-&gt;ndpi_struct, flow_to_process-&gt;guessed_protocol.proto.app_protocol),\n\t     ndpi_category_get_name(workflow-&gt;ndpi_struct, flow_to_process-&gt;guessed_protocol.category));\n    } else {\n      printf(\"&#91;%8llu, %d, %4d]&#91;FLOW NOT CLASSIFIED]\\n\",\n\t     workflow-&gt;packets_captured, reader_thread-&gt;array_index, flow_to_process-&gt;flow_id);\n    }\n  }\n\n  flow_to_process-&gt;detected_l7_protocol =\n    ndpi_detection_process_packet(workflow-&gt;ndpi_struct, flow_to_process-&gt;ndpi_flow,\n\t\t\t\t  ip != NULL ? (uint8_t *)ip : (uint8_t *)ip6,\n\t\t\t\t  ip_size, time_ms, NULL);\n\n  if (ndpi_is_protocol_detected(flow_to_process-&gt;detected_l7_protocol) != 0 &amp;&amp;\n      flow_to_process-&gt;detection_completed == 0)\n    {\n      if (flow_to_process-&gt;detected_l7_protocol.proto.master_protocol != NDPI_PROTOCOL_UNKNOWN ||\n          flow_to_process-&gt;detected_l7_protocol.proto.app_protocol != NDPI_PROTOCOL_UNKNOWN)\n      {\n        flow_to_process-&gt;detection_completed = 1;\n        workflow-&gt;detected_flow_protocols++;\n\n        printf(\"&#91;%8llu, %d, %4d]&#91;DETECTED] protocol: %s | app protocol: %s | category: %s\\n\",\n\t       workflow-&gt;packets_captured,\n\t       reader_thread-&gt;array_index,\n\t       flow_to_process-&gt;flow_id,\n\t       ndpi_get_proto_name(workflow-&gt;ndpi_struct, flow_to_process-&gt;detected_l7_protocol.proto.master_protocol),\n\t       ndpi_get_proto_name(workflow-&gt;ndpi_struct, flow_to_process-&gt;detected_l7_protocol.proto.app_protocol),\n\t       ndpi_category_get_name(workflow-&gt;ndpi_struct, flow_to_process-&gt;detected_l7_protocol.category));\n      }\n    }\n\n  if (flow_to_process-&gt;ndpi_flow-&gt;num_extra_packets_checked &lt;=\n      flow_to_process-&gt;ndpi_flow-&gt;max_extra_packets_to_check)\n    {\n      \/*\n       * Your business logic starts here.\n       *\n       * This example does print some information about\n       * TLS client and server hellos if available.\n       *\n       * You could also use nDPI's built-in json serialization\n       * and send it to a high-level application for further processing.\n       *\n       * EoE - End of Example\n       *\/\n\n      if (flow_to_process-&gt;flow_info_printed == 0)\n      {\n        char const * const flow_info = ndpi_get_flow_info(flow_to_process-&gt;ndpi_flow, &amp;flow_to_process-&gt;detected_l7_protocol);\n        if (flow_info != NULL)\n        {\n          printf(\"&#91;%8llu, %d, %4d] info: %s\\n\",\n            workflow-&gt;packets_captured,\n            reader_thread-&gt;array_index,\n            flow_to_process-&gt;flow_id,\n            flow_info);\n          flow_to_process-&gt;flow_info_printed = 1;\n        }\n      }\n\n      if (flow_to_process-&gt;detected_l7_protocol.proto.master_protocol == NDPI_PROTOCOL_TLS ||\n\t  flow_to_process-&gt;detected_l7_protocol.proto.app_protocol == NDPI_PROTOCOL_TLS)\n        {\n\t  if (flow_to_process-&gt;tls_client_hello_seen == 0 &amp;&amp;\n\t      flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.client_hello_processed != 0)\n            {\n\t      uint8_t unknown_tls_version = 0;\n\t      char buf_ver&#91;16];\n\t      printf(\"&#91;%8llu, %d, %4d]&#91;TLS-CLIENT-HELLO] version: %s | sni: %s | (advertised) ALPNs: %s\\n\",\n\t\t     workflow-&gt;packets_captured,\n\t\t     reader_thread-&gt;array_index,\n\t\t     flow_to_process-&gt;flow_id,\n\t\t     ndpi_ssl_version2str(buf_ver, sizeof(buf_ver),\n\t\t\t\t\t  flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.ssl_version,\n\t\t\t\t\t  &amp;unknown_tls_version),\n\t\t     flow_to_process-&gt;ndpi_flow-&gt;host_server_name,\n\t\t     (flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.advertised_alpns != NULL ?\n\t\t      flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.advertised_alpns : \"-\"));\n\t      flow_to_process-&gt;tls_client_hello_seen = 1;\n            }\n\t  if (flow_to_process-&gt;tls_server_hello_seen == 0 &amp;&amp;\n\t      flow_to_process-&gt;ndpi_flow-&gt;tls_quic.certificate_processed != 0)\n            {\n\t      uint8_t unknown_tls_version = 0;\n\t      char buf_ver&#91;16];\n\t      printf(\"&#91;%8llu, %d, %4d]&#91;TLS-SERVER-HELLO] version: %s | common-name(s): %.*s | \"\n\t\t     \"issuer: %s | subject: %s\\n\",\n\t\t     workflow-&gt;packets_captured,\n\t\t     reader_thread-&gt;array_index,\n\t\t     flow_to_process-&gt;flow_id,\n\t\t     ndpi_ssl_version2str(buf_ver, sizeof(buf_ver),\n\t\t\t\t\t  flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.ssl_version,\n\t\t\t\t\t  &amp;unknown_tls_version),\n\t\t     (flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.server_names_len == 0 ?\n\t\t      1 : flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.server_names_len),\n\t\t     (flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.server_names == NULL ?\n\t\t      \"-\" : flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.server_names),\n\t\t     (flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.issuerDN != NULL ?\n\t\t      flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.issuerDN : \"-\"),\n\t\t     (flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.subjectDN != NULL ?\n\t\t      flow_to_process-&gt;ndpi_flow-&gt;protos.tls_quic.subjectDN : \"-\"));\n\t      flow_to_process-&gt;tls_server_hello_seen = 1;\n            }\n        }\n    }\n}\n\nstatic void run_pcap_loop(struct nDPI_reader_thread const * const reader_thread)\n{\n  if (reader_thread-&gt;workflow != NULL &amp;&amp;\n      reader_thread-&gt;workflow-&gt;pcap_handle != NULL) {\n\n    if (pcap_loop(reader_thread-&gt;workflow-&gt;pcap_handle, -1,\n\t\t  &amp;ndpi_process_packet, (uint8_t *)reader_thread) == PCAP_ERROR) {\n\n      fprintf(stderr, \"Error while reading pcap file: '%s'\\n\",\n\t      pcap_geterr(reader_thread-&gt;workflow-&gt;pcap_handle));\n      __sync_fetch_and_add(&amp;reader_thread-&gt;workflow-&gt;error_or_eof, 1);\n    }\n  }\n}\n\nstatic void break_pcap_loop(struct nDPI_reader_thread * const reader_thread)\n{\n  if (reader_thread-&gt;workflow != NULL &amp;&amp;\n      reader_thread-&gt;workflow-&gt;pcap_handle != NULL)\n    {\n      pcap_breakloop(reader_thread-&gt;workflow-&gt;pcap_handle);\n    }\n}\n\nstatic void * processing_thread(void * const ndpi_thread_arg)\n{\n  struct nDPI_reader_thread const * const reader_thread =\n    (struct nDPI_reader_thread *)ndpi_thread_arg;\n\n  printf(\"Starting Thread %d\\n\", reader_thread-&gt;array_index);\n  run_pcap_loop(reader_thread);\n  __sync_fetch_and_add(&amp;reader_thread-&gt;workflow-&gt;error_or_eof, 1);\n  return NULL;\n}\n\nstatic int processing_threads_error_or_eof(void)\n{\n  int i;\n\n  for (i = 0; i &lt; reader_thread_count; ++i) {\n    if (__sync_fetch_and_add(&amp;reader_threads&#91;i].workflow-&gt;error_or_eof, 0) == 0) {\n      return 0;\n    }\n  }\n  return 1;\n}\n\nstatic int start_reader_threads(void)\n{\n  int i;\n\n#ifndef WIN32\n  sigset_t thread_signal_set, old_signal_set;\n\n  sigfillset(&amp;thread_signal_set);\n  sigdelset(&amp;thread_signal_set, SIGINT);\n  sigdelset(&amp;thread_signal_set, SIGTERM);\n  if (pthread_sigmask(SIG_BLOCK, &amp;thread_signal_set, &amp;old_signal_set) != 0) {\n    fprintf(stderr, \"pthread_sigmask: %s\\n\", strerror(errno));\n    return 1;\n  }\n#endif\n\n  for (i = 0; i &lt; reader_thread_count; ++i) {\n    reader_threads&#91;i].array_index = i;\n\n    if (reader_threads&#91;i].workflow == NULL) {\n      \/* no more threads should be started *\/\n      break;\n    }\n\n    if (pthread_create(&amp;reader_threads&#91;i].thread_id, NULL,\n\t\t       processing_thread, &amp;reader_threads&#91;i]) != 0)\n      {\n\tfprintf(stderr, \"pthread_create: %s\\n\", strerror(errno));\n\treturn 1;\n      }\n  }\n\n  if (pthread_sigmask(SIG_BLOCK, &amp;old_signal_set, NULL) != 0) {\n    fprintf(stderr, \"pthread_sigmask: %s\\n\", strerror(errno));\n    return 1;\n  }\n\n  return 0;\n}\n\nstatic int stop_reader_threads(void)\n{\n  int i;\n  unsigned long long int total_packets_captured = 0;\n  unsigned long long int total_packets_processed = 0;\n  unsigned long long int total_l4_data_len = 0;\n  unsigned long long int total_flows_captured = 0;\n  unsigned long long int total_flows_idle = 0;\n  unsigned long long int total_flows_detected = 0;\n\n  for (i = 0; i &lt; reader_thread_count; ++i) {\n    break_pcap_loop(&amp;reader_threads&#91;i]);\n  }\n\n  printf(\"------------------------------------ Stopping reader threads\\n\");\n\n  for (i = 0; i &lt; reader_thread_count; ++i) {\n    if (reader_threads&#91;i].workflow == NULL) {\n      continue;\n    }\n\n    if (pthread_join(reader_threads&#91;i].thread_id, NULL) != 0) {\n      fprintf(stderr, \"pthread_join: %s\\n\", strerror(errno));\n    }\n\n    total_packets_processed += reader_threads&#91;i].workflow-&gt;packets_processed;\n    total_l4_data_len += reader_threads&#91;i].workflow-&gt;total_l4_data_len;\n    total_flows_captured += reader_threads&#91;i].workflow-&gt;total_active_flows;\n    total_flows_idle += reader_threads&#91;i].workflow-&gt;total_idle_flows;\n    total_flows_detected += reader_threads&#91;i].workflow-&gt;detected_flow_protocols;\n\n    printf(\"Stopping Thread %d, processed %10llu packets, %12llu bytes, total flows: %8llu, \"\n\t   \"idle flows: %8llu, detected flows: %8llu\\n\",\n\t   reader_threads&#91;i].array_index, reader_threads&#91;i].workflow-&gt;packets_processed,\n\t   reader_threads&#91;i].workflow-&gt;total_l4_data_len, reader_threads&#91;i].workflow-&gt;total_active_flows,\n\t   reader_threads&#91;i].workflow-&gt;total_idle_flows, reader_threads&#91;i].workflow-&gt;detected_flow_protocols);\n  }\n\n  \/* total packets captured: same value for all threads as packet2thread distribution happens later *\/\n  total_packets_captured = reader_threads&#91;0].workflow-&gt;packets_captured;\n\n  for (i = 0; i &lt; reader_thread_count; ++i) {\n    if (reader_threads&#91;i].workflow == NULL) {\n      continue;\n    }\n\n    free_workflow(&amp;reader_threads&#91;i].workflow);\n  }\n\n  printf(\"Total packets captured.: %llu\\n\", total_packets_captured);\n  printf(\"Total packets processed: %llu\\n\", total_packets_processed);\n  printf(\"Total layer4 data size.: %llu\\n\", total_l4_data_len);\n  printf(\"Total flows captured...: %llu\\n\", total_flows_captured);\n  printf(\"Total flows timed out..: %llu\\n\", total_flows_idle);\n  printf(\"Total flows detected...: %llu\\n\", total_flows_detected);\n\n  return 0;\n}\n\nstatic void sighandler(int signum)\n{\n  fprintf(stderr, \"Received SIGNAL %d\\n\", signum);\n\n  if (__sync_fetch_and_add(&amp;main_thread_shutdown, 0) == 0) {\n    __sync_fetch_and_add(&amp;main_thread_shutdown, 1);\n  } else {\n    fprintf(stderr, \"Reader threads are already shutting down, please be patient.\\n\");\n  }\n}\n\nint main(int argc, char ** argv)\n{\n  if (argc == 0) {\n    printf(\"usage: ndpiSimpleIntegration &lt;device name&gt;\\n\");\n    return 1;\n  }\n  \n  printf(\"usage: %s &#91;PCAP-FILE-OR-INTERFACE]\\n\"\n\t \"----------------------------------\\n\"\n\t \"nDPI version: %s\\n\"\n\t \" API version: %u\\n\"\n\t \"libgcrypt...: %s\\n\"\n\t \"----------------------------------\\n\",\n\t argv&#91;0],\n\t ndpi_revision(), ndpi_get_api_version(),\n\t (ndpi_get_gcrypt_version() == NULL ? \"-\" : ndpi_get_gcrypt_version()));\n\n  if (setup_reader_threads((argc &gt;= 2 ? argv&#91;1] : NULL)) != 0) {\n    fprintf(stderr, \"%s: setup_reader_threads failed\\n\", argv&#91;0]);\n    return 1;\n  }\n\n  if (start_reader_threads() != 0) {\n    fprintf(stderr, \"%s: start_reader_threads\\n\", argv&#91;0]);\n    return 1;\n  }\n\n  signal(SIGINT, sighandler);\n  signal(SIGTERM, sighandler);\n  while (__sync_fetch_and_add(&amp;main_thread_shutdown, 0) == 0 &amp;&amp; processing_threads_error_or_eof() == 0) {\n    sleep(1);\n  }\n\n  if (stop_reader_threads() != 0) {\n    fprintf(stderr, \"%s: stop_reader_threads\\n\", argv&#91;0]);\n    return 1;\n  }\n\n  return 0;\n}\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Podcast: explore FCC Enforcement Applications of nDPI Welcome to another episode of our podcast, where we explore the tools and code that power modern network analysis! Today, we\u2019re taking a close look at\u00a0ndpiSimpleIntegration.c, a comprehensive example from the nDPI project\u2014a popular open-source deep packet inspection toolkit. ndpiSimpleIntegration.c \u2013 Network Traffic Analysis in Action What is&hellip;&nbsp;<a href=\"https:\/\/172-234-197-23.ip.linodeusercontent.com\/?p=2001\" rel=\"bookmark\"><span class=\"screen-reader-text\">nDPI Network Traffic Analysis<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":2003,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","footnotes":""},"categories":[14,7],"tags":[],"class_list":["post-2001","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-podcast","category-the-truben-show"],"_links":{"self":[{"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=\/wp\/v2\/posts\/2001","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2001"}],"version-history":[{"count":5,"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=\/wp\/v2\/posts\/2001\/revisions"}],"predecessor-version":[{"id":2010,"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=\/wp\/v2\/posts\/2001\/revisions\/2010"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=\/wp\/v2\/media\/2003"}],"wp:attachment":[{"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2001"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2001"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2001"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}