What is Netflow?
Netflow by Cisco which is now an open standard is the unknown networking tool that can do wonders for network monitoring.
If you are into network management and wish to learn how your network is functioning then netflow is the go to tool.
It is freely available on all Linux machines and you can understand a lot about Internet and your own network with it. The concept of a flow does not exist for UDP and ICMP protocols but Netflow still records these packet exchanges as also flows like a TCP flow.
You can have network flows in either direction, depending on if it is an upload or a download. You have various protocols tunneled using ssh, all resulting in flows between two endpoints based on
Source port
Source IP
Destination port
Destination IP
Timestamp
Duration of flow
Packets
Remember each flow is unidirectional. So every connection has flows in both directions , so you can expect two entries in netflow output.
How does it work?
A flow is a stream of packets that share the same characteristics like source/destination port, source/destination address, protocol, type, service marking(QoS), etc.
For each of the flows, NetFlow will track the number of packets sent, bytes sent, packet sizes and more. You can configure your router to keep track of all flows and then export them to a central server where we analyze our flows.
Netflow first came into existence in 1996 and is now an open standard. It is also known as IPFIX in latter versions.
What is softflowd?
On Linux systems several tools support netflow. You have flow-tools, nfdump and more. softflowd by OpenBSD developer Damien Miller is my choice.
What is a netflow collector?
Netflow flows are all collected and stored into a database using UDP or SCTP. The protocol used by netflow for sending the flow data itself is quite involved. This is due to the huge amount of packet traffic we now see on important choke points on Internet.
The collector can store the flows for later retrieval and analysis. A flow expires based on timers. In case of UDP and ICMP there is no notion of flow anyway.
What is a netflow exporter?
The exporter is the node collecting all the flows from a network interface. SNMP has notion of ingress and egress interface. Netflow does include this information along with even TCP flags or Autonomous System number depending on what flow we are measuring.
It also has the IP protocol number information. Netflow exporter node is supposed to be efficient in processing flows as packets are always flowing through busy nodes.
What is a netflow analyzer?
The analyzer node has no workload as the flow information is sitting in collector node and you can create graphs or build tables or insert flows into a database offline.
You could also combine these two functions into one node and perform flow analysis on the node that sees the traffic.
How is it stored?
Netflow and pcap are similar. And netflow flows can be built from pcap as well. You need a lot of storage since the flows in a packet switched network can be quite voluminous.
So flow records are typically compressed for later analysis.
Can it work like pcap?
Just like tcpdump or wireshark netflow also has to look at packets at wire speeds and this can be pretty overwhelming. We have high speed Internet these days and overloading critical network infrastructure for measurements and analysis is not a good idea.
Where is it used?
Netflow records can be a looking glass to understand your network. Today all media that flows over from one place to another constitutes a flow be it some voice over RTP which uses UDP or a video call or a video stream.
Netflow records can give you a really good idea of where your network resources go towards giving you a rich end user experience.
Netflow is the standard way to make network based measurements. SNMP MIB can give you other data on nodes and manufacturers give you that info. But packet flows can only be mananged by netflow.
How can you benefit?
Nowadays Linux conntrack feature in kernel supports netflow. This is supported by all commercial router vendors. OpenWRT firmware running on low powered devices all run Linux. So they also support netflow.
The comprehensive 30,000’ view given by flow records is really nice.
Conclusion
We have seen how Cisco Netflow v10 which is now an open standard can be used to understand your network. It gives a very good overview of your network usage without going through the details of a wireshark network grab.
Also for heavily network intensive projects you can split the flow records processing into exporter, collector and analyzer.
Answer
What is meant by leap second?
Leap seconds are extra seconds inserted into time keeping NTP stratum one servers. This is due to irregularities and slowdowns in earth’s rotation.
The 2004 Indian Ocean earthquake is thought to have shortened it by 2.68 microseconds.
Usually NTP servers get high precision time from Cesium crystals.
Leap second is similar to leap year in time keeping. We may lose a second after a while and we insert it for keeping time accurate. NTP was my first significant coding project. Computers can get very high accuracy by running NTP daemons.
Question
What does the tool jq do?
Feedback
If you feel like covering a certain topic of interest to you then you can send a note and I shall feature it. All it takes is a reply to this mail.