Pfmatch, a packet filtering language embedded in Lua

3 July 2015 11:05 AM (lua | pflua | bpf | pflang | igalia | snabb | compilers | dsl | edsl)

Greets, hackers! I just finished implementing a little embedded language in Lua and wanted to share it with you. First, a bit about the language, then some notes on how it works with Lua to reach the high performance targets of Snabb Switch.

the pfmatch language

Pfmatch is a language designed for filtering, classifying, and dispatching network packets in Lua. Pfmatch is built on the well-known pflang packet filtering language, using the fast pflua compiler for LuaJIT.

Here's an example of a simple pfmatch program that just divides up packets depending on whether they are TCP, UDP, or something else:

match {
   tcp => handle_tcp
   udp => handle_udp
   otherwise => handle_other

Unlike pflang filters written for such tools as tcpdump, a pfmatch program can dispatch packets to multiple handlers, potentially destructuring them along the way. In contrast, a pflang filter can only say "yes" or "no" on a packet.

Here's a more complicated example that passes all non-IP traffic, drops all IP traffic that is not going to or coming from certain IP addresses, and calls a handler on the rest of the traffic.

match {
   not ip => forward
   ip src => incoming_ip
   ip dst => outgoing_ip
   otherwise => drop

In the example above, the handlers after the arrows (forward, incoming_ip, outgoing_ip, and drop) are Lua functions. The part before the arrow (not ip and so on) is a pflang expression. If the pflang expression matches, its handler will be called with two arguments: the packet data and the length. For example, if the not ip pflang expression is true on the packet, the forward handler will be called.

It's also possible for the handler of an expression to be a sub-match:

match {
   not ip => forward
   ip src => {
      tcp => incoming_tcp(&ip[0], &tcp[0])
      udp => incoming_udp(&ip[0], &ucp[0])
      otherwise => incoming_ip(&ip[0])
   ip dst => {
      tcp => outgoing_tcp(&ip[0], &tcp[0])
      udp => outgoing_udp(&ip[0], &ucp[0])
      otherwise => outgoing_ip(&ip[0])
   otherwise => drop

As you can see, the handlers can also have additional arguments, beyond the implicit packet data and length. In the above example, if not ip doesn't match, then ip src matches, then tcp matches, then the incoming_tcp function will be called with four arguments: the packet data as a uint8_t* pointer, its length in bytes, the offset of byte 0 of the IP header, and the offset of byte 0 of the TCP header. An argument to a handler can be any arithmetic expression of pflang; in this case &ip[0] is actually an extension. More on that later. For language lawyers, check the syntax and semantics over in our source repo.

Thanks especially to my colleague Katerina Barone-Adesi for long backs and forths about the language design; they really made it better. Fistbump!

pfmatch and lua

The challenge of designing pfmatch is to gain expressiveness, compared to writing filters by hand, while not endangering the performance targets of Pflua and Snabb Switch. These days Snabb is on target to give ASIC-driven network appliances a run for their money, so anything we come up with cannot sacrifice speed.

In practice what this means is compile, don't interpret. Using the pflua compiler allows us to generalize the good performance that we have gotten on pflang expressions to a multiple-dispatch scenario. It's a pretty straightword strategy. Naturally though, the interface with Lua is more complex now, so to understand the performance we should understand the interaction with Lua.

How does one make two languages interoperate, anyway? With pflang it's pretty clear: you compile pflang to a Lua function, and call the Lua function to match on packets. It returns true or false. It's a thin interface. Indeed with pflang and pflua you could just match the clauses in order:

not_ip = pf.compile('not ip')
incoming = pf.compile('ip src')
outgoing = pf.compile('ip dst')

function handle(packet, len)
   if not_ip(packet, len) then return forward(packet, len)
   elseif incoming(packet, len) then return incoming_ip(packet, len)
   elseif outgoing(packet, len) then return outgoing_ip(packet, len)
   else return drop(packet, len) end

But not only is this tedious, you don't get easy access to the packet itself, and you're missing out on opportunities for optimization. For example, if the packet fails the not_ip check, we don't need to check if it's an IP packet in the incoming check. Compiling a pfmatch program takes advantage of pflua's optimizer to produce good code for the match expression as a whole.

If this were Scheme I would make the right-hand side of an arrow be an expression and implement pfmatch as a macro; see Racket's match documentation for an example. In Lua or other languages that's harder to do; you would have to parse Lua, and it's not clear which parts of the production as a whole are the host language (Lua) and which are the embedded language (pfmatch).

Instead, I think embedding host language snippets by function name is a fine solution. It seems fairly clear that incoming_ip, for example, is some kind of function. It's easy to parse identifiers in an embedded language, both for humans and for programs, so that takes away a lot of implementation headache and cognitive overhead.

We are left with a few problems: how to map names to functions, what to do about the return value of match expressions, and how to tie it all together in the host language. Again, if this were Scheme then I'd use macros to embed expressions into the pfmatch term, and their names would be scoped into whatever environment the match term was defined. In Lua, the best way to implement a name/value mapping is with a table. So we have:

local handlers = {
   forward = function(data, len)
   drop = function(data, len)
   incoming_ip = function(data, len)
   outgoing_ip = function(data, len)

Then we will pass the handlers table to the matcher function, and the matcher function will call the handlers by name. LuaJIT will mostly take care of the overhead of the table dispatch. We compile the filter like this:

local match = require('pf.match')

local dispatcher = match.compile([[match {
   not ip => forward
   ip src => incoming_ip
   ip dst => outgoing_ip
   otherwise => drop

To use it, you just invoke the dispatcher with the handlers, data, and length, and the return value is whatever the handler returns. Here let's assume it's a boolean.

function loop(self)
   local i, o = self.input.input, self.output.output
   while not link.empty() do
      local pkt = link.receive(i)
      if dispatcher(handlers,, pkt.length) then
         link.transmit(o, pkt)

Finally, we're ready for an example of a compiled matcher function. Here's what pflua does with the match expression above:

local cast = require("ffi").cast
return function(self,P,length)
   if length < 14 then return self.forward(P, len) end
   if cast("uint16_t*", P+12)[0] ~= 8 then return self.forward(P, len) end
   if length < 34 then return self.drop(P, len) end
   if P[23] ~= 6 then return self.drop(P, len) end
   if cast("uint32_t*", P+26)[0] == 67305985 then return self.incoming_ip(P, len) end
   if cast("uint32_t*", P+30)[0] == 134678021 then return self.outgoing_ip(P, len) end
   return self.drop(P, len)

The result is a pretty good dispatcher. There are always things to improve, but it's likely that the function above is better than what you would write by hand, and it will continue to get better as pflua improves.

Getting back to what I mentioned earlier, when we write filtering code by hand, we inevitably end up writing interpreters for some kind of filtering language. Network functions are essentially linguistic in nature: static appliances are no good because network topologies change, and people want solutions that reflect their problems. Usually this means embedding an interpreter for some embedded language, for example BPF bytecode or iptables rules. Using pflua and pfmatch expressions, we can instead compile a filter suited directly for the problem at hand -- and while we're at it, we can forget about worrying about pesky offsets, constants, and bit-shifts.


I'm optimistic about pfmatch or something like it being a success, but there are some challenges too.

One challenge is that pflang is pretty weird. For example, attempting to access ip[100] will abort a filter immediately on a packet that is less than 100 bytes long, not including L2 encapsulation. It's wonky semantics, and in the context of pfmatch, aborting the entire pfmatch program would obviously be the wrong thing. That would abort too much. Instead it should probably just fail the pflang test in which that packet access appears. To this end, in pfmatch we turn those aborts into local expression match failures. However, this leads to an inconsistency with pflang. For example in (ip[100000] == 0 or (1==1)), instead of ip[100000] causing the whole pflang match to fail, it just causes the local test to fail. This leaves us with 1==1, which passes. We abort too little.

This inconsistency is probably a bug. We want people to be able to test clauses with vanilla pflang expressions, and have the result match the pfmatch behavior. Due to limitations in some of pflua's intermediate languages, it's likely to persist for a while. It is the only inconsistency that I know of, though.

Pflang is also underpowered in many ways. It has terrible IPv6 support; for example, tcp[0] only matches IPv4 packets, and at least as implemented in libpcap, most payload access on IPv6 packets does the wrong thing regarding chained extension headers. There is no facility in the language for binding names to intermediate results, there is no linguistic facility for talking about fragmentation, no ability to address IP source and destination addresses in arithmetic expressions by name, and so on. We can solve these in pflua with extensions to the language, but that introduces incompatibilities with pflang.

You might wonder why to stick with pflang, after all of this. If this is you, Juho Snellman wrote a great article on this topic, just for you: What's wrong with pcap filters.

Pflua's optimizer has mostly helped us, but there have been places where it could be more helpful. When compiling just one expression, you can often end up figuring out which branches are dead-ends, which helps the rest of the optimization to proceed. With more than one successful branch, we had to make a few improvements to the optimizer to actually get decent results. We also had to relax one restriction on the optimizer: usually we only permit transformations that make the code smaller. This way we know we're going in the right direction and will eventually terminate. However because of reasons™ we did decide to allow tail calls to be duplicated, so instead of having just one place in the match function that tail-calls a handler, you can end up with multiple calls. I suspect using a tracing compiler will largely make this moot, as control-flow splits effectively lead to trace duplication anyway, and making sure control-flow joins later doesn't effectively counter that. Still, I suspect that the resulting trace shape will rejoin only at the loop head, instead of in some intermediate point, which is probably OK.


With all of these concerns, is pfmatch still a win? Yes, probably! We're going to start using it when building Snabb apps, and will see how it goes. We'll probably end up adding a few more pflang extensions before we're done. If it's something you're in to, snabb-devel is the place to try it out, and see you on the bug tracker. Happy packet hacking!

78 responses

  1. Galaxy S9 says:

    Samsung has become predictable when it comes to product launching. Now it is time for Samsung Galaxy S9 and Galaxy S9 Price. Samsung Galaxy S9 release ..

  2. website says:

    Great work! I think it will be useful for app developers and code-writers.

  3. Samsung Galaxy S9 says:

    Thanks for the nice post.
    Visit my website for the upcoming samsung galaxy s9 release date and price.

  4. Ashish Kumar says:

    I suspect using a tracing compiler will largely make this moot, as control-flow splits effectively lead to trace duplication anyway, and making sure control-flow joins later doesn't effectively counter that.

  5. clash of clans online hack says:

    Clash of clans game is played all over the world and is very popular as it provides a real sense of online gaming.

  6. how to change user password windows 10 says:

    This is for ps thanks for the site is the homepage the great star.

  7. Assignment Help London says:

    Running packet filtering language on multiple languages is a different filter which is based on Network functions linguistically.

  8. Custom Essay Writing Service says:

    The second part is yet to be composed, it will come in the following blog entry, however it will most likely have a more "current" composition style.

  9. Pay Someone To Do My Essay says:

    I'm constructive approximately pfmatch or something like it being a success, however there are some challenges too.

  10. free robux says:

    Hey!! find free roblox robux for me.

  11. essay help says:

    The challenge of designing pfmatch is to gain expressiveness, compared to writing filters by hand, while not endangering the performance targets of Pflua and Snabb Switch. in this paragraph i read many new things which is really good and which i have never noticed before.

  12. phd dissertation writers says:

    If match, a packet filtering language embedded in Lua. 31 months ago embedded Lua networking · Get it on Google Play. Created by Matt Mastracci (Matthew - at - This entire project is open source. Also, see Starbase - visual Unicode database.

  13. walmart credit card login says:

    Walmart, the most profitable retailer in the USA, also offers two credit cards for its customers. Synchrony Bank issues both the cards. Hence, any card related issues; Synchrony bank will be providing customer service.

  14. Assignment says:

    Having with general background on your topic is also necessary to write a good history paper.

  15. discover credit card login says:

    With the changing trends and advancements in technology; credit cards have now become an indistinguishable part of the different day

  16. Best Assignment Writing Service says:

    Running bundle sifting dialect on numerous dialects is an alternate channel which depends on Network works phonetically.

  17. Solusi Alternatif Untuk Mengatasi Kutil Kelamin says:

    The mission of the Young Scientist Lab is to foster a new generation of scientists who are inspired to improve the world with science. For teachers, students and parents seeking a place to explore the world of scientific innovations and opportunities.

  18. Civil Engineering Assignment Help Service says:

    The mission of the Young Scientist Lab is to foster a new generation of scientists who are inspired to improve the world with science. For teachers, students and parents seeking a place to explore the world of scientific innovations and opportunities.

  19. Cheap Essay Writing Service says:

    The check of designing of the match is to select up quality, contrasted with composing channels by hand, whereas not jeopardizing the execution focuses of Pflua and Snabb Switch. during this section, I browse varied new things that is okay and that I even have ne'er taken note.

  20. High Da Guest Posting Website List says:

    This is a great star for PS because of the website that is a home page. Your article is for me, and you share this great and useful information here.

  21. foil a car dubai says:

    When aggregating only one articulation, you can regularly wind up making sense of which branches are deadlocks, which helps whatever remains of the streamlining to continue.

  22. Best Essay Writing Service says:

    The check of coming up with of the match is to decide on up quality, appeared otherwise in relevancy creating channels by hand, the' not risking the execution focal points of Pflua and Snabb Switch. Amid this space, I examine fluctuated new things that's alright which I even have never discovered.

  23. Delete browsing history says:

    When you surf on the internet delete your search history and browsing data is a must. You may clear all your activity before leaving your computer, smartphone or tablet.

  24. searchhistory says:

    The Search History is a list of all our past searches in the main search engines like google, bing or duck duck go.

  25. University Essay Help says:

    I speculate utilizing a following compiler will to a great extent make this unsettled, as control-stream parts successfully prompt follow duplication at any rate, and ensuring control-stream joins later doesn't adequately counter that.

  26. UK Dissertation Help says:

    Have you had a tendency that you don't have the correct stuff anticipated that would finish the paper and the due date is drawing nearer? Do you feel like as of now is that day. Stress not my sweet amigo as we are here, British Dissertation Writers, as the unrivaled recipients of insightful substance creating, willing to help you out. Our dissertation help will flabbergast you.

  27. says:

    useful information on topics that plenty are interested on for this wonderful post.Admiring the time and effort you put into your b!..

  28. sound not working says:

    Here we have free support for those who wants free windows support.

  29. Assignment Help says:

    A high-standard post with all imperative information about Assignment Help UK services. Looking forward to availing the premium services.

  30. Assignment Help says:

    A high-standard post with all imperative information about Assignment Help UK services. Looking forward to availing the premium services.

  31. Online Dissertation Help in the UK says:

    A lot of education person will find this valuable. It can not be right but they will benefit and useful for education from its Online Dissertation Help in the UK

  32. Perth assignment help says:

    perth assignment help great quality and the service provided is up to date and easily relatable.

  33. used cat wheel loader 938gii-aa003640u for sale says:

    Thanks so much for a great post. I'd like to know more about these topics and hope that I can receive more insight into this topic.


    You should try to keep your article short. If you want to finish in lessen time. Make sure worlds limit should not exceed more than 500 words. Try to add only relevant data and information. If you feel that you are stuck, you can take assignment help online to get a proper idea. There are several web services which can help you to find out a suitable solution to your problems.

  35. ant man sky go says:

    . Not only do you get unlimited movie downloads you also get music. Both the music and movie downloads are from the largest lists of selections I have seen.

  36. says:

    Students who use some other essay writing services often walk away disappointed. Many times, they receive an essay or paper that is so poorly written, it cannot possibly be turned in. Such an outcome will never happen at Trust My Paper. When you ask us to write an essay, you will have an entirely different experience, and you will receive a wholly original and perfectly written piece by the deadline. This is our guarantee to you and every single customer who comes to us for help. If you are looking for information about trustmypaper review, welcome to Scamfigter!

  37. cv writing service says:

    Many resumes are discarded because they fail to capture the interest of the recruiter during a brief scan. Therefore, jobseekers are advised to restrict themselves to no more than 1-2 sides when compiling their resume. . If you need "cv writing service", welcome to

  38. unit 1 business environment assignment sample ikea says:

    My name is Michael Smith and This is a very awesome post. Thanks for sharing this.

  39. transport routier Guadeloupe says:

    It is fantastic post. I'll be waiting for more posts

  40. perdisco assignment help says:

    this is roxy smith.
    this is a nice post, thanks for sharing.
    Get the best perdisco assignment help in Australia.

  41. Best Essay Writers UK says:

    The Pixel Farm has announced what it calls the giant leap forward in tracking software, PFMatchit. PFMatchit is 64 bit, node based, and cheap listed at a price of around $700.00 available sometime in August

  42. Tips for Success at College says:

    Well look, even if that tool existed I would not recommend it, because in the end we will have spam in all those groups and in those groups they will sign you as you always publish in an automatic way and in a non-personal way. So I would really define those 50 groups, what different characteristics they have, start the conversation differently and on different days because there will be people who are also in many of those groups and it will not seem good if you publish the same in all sites at once. So, rather, differentiate the strategy and also differentiate the days in the publication times.

  43. Assignment Help says:

    An unmatched and nonpareil post i have ever seen. The content is so appealing that it has created an impulse to avail Assignment Help Singapore services.

  44. Dev Chaudhary says:

    The website is working in the field of Kids Education and Entertainment. Story characters which teach Moral Values and learning techniques to kids through Stories, Moral lessons, Inspirational Stories, Fairy Tales, Fables, Love Stories, Bedtime Stories. There is a need to give today's generation moral values and an easy way to learn things.

  45. thesis statements says:

    Nice post, it was very informative and I learnt a lot of new things. By the way, talking about useful things. Recently I found a thesis generator on one of the students’ websites. It makes thesis statements and is absolutely free. Maybe, somebody will find it helpful.

  46. Kryptowaluty says:

    I am searching for some good site for learning.

  47. Assignment Help UAE says:

    Best Assignment Help UAE in UAE.

  48. 123helpme says:

    Hi! If you need help with your university assignments, visit our writing ny and get some help.

  49. Essay Mafia Google+ says:

    The main aim of Wing Chun Kung Fu is to make you have freer and more relaxed movement. This is achieved through the five principles of center-line,

  50. Online Assignment Writing Services UK says:

    I estimate using the following compiler will, all things considered, make this disrupted, as control-stream parts effectively incite pursue duplication at any rate, and guaranteeing control-stream joins later doesn't sufficiently counter that.

  51. Cheap Assignment Writing Service says:

    Pleasant post, it was extremely instructive and I took in a ton of new things. Coincidentally, discussing valuable things. As of late I found a proposition generator on one of the understudies' sites. It puts forth proposal expressions and is totally free. Possibly, someone will think that its supportive.

  52. law assignment says:

    We have been into the education industry from past few years. We are Go Assignment Help which offers online assignment help, assignment help in Melbourne for students and learning professionals. Take the advantage of nursing assignment help or all assignment help only at our one-stop destination for assignment needs.

  53. Pakistani portoro black and gold marble says:

    Hi my friend! I would like to articulate that this piece of writing is overwhelming, immense written and comprise roughly each and every one remarkable info. I would like to peer added posts like this.

  54. Evelyn W. Minnick says:

    Are you clueless about difficult assignment worksheets? Are the most of your time wasted in struggling with only one subject? Do not spend your time worrying. Act fast and call for Assignment Help online from us at Our panel of professional educators assists students in constructing scoring answers and in acquiring extra credits.

  55. colineddie says:


  56. colineddie05 says:


  57. career episode sample electrical engineer says:

    Wow, fantastic blog layout! How long have you been blogging for?
    you made blogging look easy. The overall look of your web site is excellent,
    let alone the content!

  58. rpl assessment sample says:

    You definitely put a brand new spin on a subject that
    has been written about for decades. Wonderful stuff,
    just excellent!

  59. says:

    Thanks for sharing. I hope it will be helpful for too many people that are searching for this topic. If you're looking for a place where you can order an essay, look no further, check our web and content checking tool.

  60. says:

    Hello there! Useful information and I'd like to be useful too and share with you the information reading which will help you to get a well-paid job because of the qualified resume you will have only just orderding it on this site.

  61. formal and informal essay says:

    Genius post. I like it so much. Writing is also my hobby. This an amazing process which helps me to be better and reach all goals.

  62. Dissertation Help Online says:

    Amazing blog with great article thanks for sharing.

  63. UK Essay Help says:

    What a great article with amazing blog thanks for sharing.

  64. how to change password on facebook says:

    Some people say that it is possible to read the messages of another person from the Facebook account only being a sender.
    If a person wants to learn how to hack someones Facebook account without any obstacles, it is enough to install a free trial
    version of anyspy app. This article is exactly what you need if you want to find out how to hack into someones Facebook

  65. Danielle says:

    Informative post. Record anything from the screen and upload it to various site by top screen recorder software.

  66. top powerful literary essay topics says:

    Hello everyone! I decided to join your conversation and add several words about writing papers. I have recently stumbled over one cool writing service that helped me a lot with writing my papers. So, I must admit that such services are very helpful!

  67. Book Critique Example says:

    What do you think about book critique? Since most experts agree that reading is one of the best ways of acquiring knowledge, tutors and professors frequently ask students to write book critiques. These tasks enhance a number of skills such as brain function and the ability to think critically.

  68. Prameya says:

    Always, I have a difficult to learn Pfmatch but this article will help me to understand bit about the Pfmatch.

  69. Information Technology Assignment Help says:

    The students who are unable to spend enough time to complete their complicated assignments may hire Online Assignment Help writers. We have professional and experienced subject experts offering you the most favourable assignment help services and ensure quality solutions and delivery in time.

  70. Homework Help Canada says:

    Assignment help additionally gives the assignment keeping in touch with the students of school assignments, paper help, article composing, and postulation composing administrations according to the college rules.

  71. Blade runner 2049 coat says:

    You have a great sense of writing I must say. Your post has those facts which are not accessible from anywhere else. It’s my humble request to u please keep writing such remarkable articles

  72. essay experts says:

    The many students who fail because of buying papers online actually fail because of plagiarism. You find that the papers they bought had not been given a personal touch. A good reason to buy papers from us is that we ensure that you get custom papers only.

  73. Computer Science Homework Help says:

    Such A Nice Post, Keep Up The Fantastic Work

  74. kitchen remodel says:

    wonderful tips you've written. I'm going to apply
    these in my regural life. Hopefully, it should increase quality of the life.

  75. lawyer for estate planning says:

    At The Amsberry Law Firm in San Antonio, we focus on helping our valued clients resolve legal concerns that have a profound impact on their lives and long-term well-being.

  76. BackupGuru says:

    Hi, I use to find difficult all the time regarding the Pfmatch.
    This article helps me, how simple to learn in easy way.
    Thanks a lot.

  77. furniture nz says:

    Hi guys.

  78. CDR report writing says:

    Many Engineering students need CDR report writing. Needcdrreport providing high-quality report writing help. You have simply contact our expert. Our expert writers are competent in crafting documents in a simple manner that makes it easy to craft a quality report.

Leave a Reply