Hyperscan is an open-source, high-performance library for regular expression matching. The library is optimized for modern CPUs, such as utilizing SIMD (Single Instruction, Multiple Data) parallelism and other hardware-specific features to accelerate pattern matching. This results in significantly faster performance compared to traditional regular expression engines when matching a large number of patterns. Its primary use cases include intrusion detection systems (IDS), intrusion prevention systems (IPS), deep packet inspection (DPI), etc. This blog post talks how to fix some errors during the build and install process of hyperscan.
1. Install dependencies
Hyperscan requires some dependencies to be installed first, including Ragel, Boost Headers, etc.
1.1 Install Ragel
$ wget http://www.colm.net/files/ragel/ragel-6.10.tar.gz
$ tar -xvf ragel-6.10.tar.gz
$ cd ragel-6.10
However, if configure directly and errors occurred as:
common.cpp:38:1: error: narrowing conversion of ‘18446744073709551615’ from ‘long unsigned int’ to ‘long long int’ [-Wnarrowing]
38 | };
| ^
common.cpp:92:1: error: narrowing conversion of ‘18446744073709551615’ from ‘long long unsigned int’ to ‘long long int’ [-Wnarrowing]
92 | };
| ^
common.cpp:120:1: error: narrowing conversion of ‘18446744073709551615’ from ‘long unsigned int’ to ‘long long int’ [-Wnarrowing]
120 | };
According to Bug 582606, it needs to add the -std=c++98 flag:
$ CXXFLAGS="-std=c++98" ./configure
Then can compile and install without errors, and ragle is installed in the default path:
$ ll /usr/local/bin
total 9.2M
-rwxr-xr-x 1 root root 1.9K Jan 18 14:20 pcap-config
-rwxr-xr-x 1 root root 2.4M Mar 20 21:28 ragel
-rwxrwxr-x 1 root root 6.8M Sep 9 2022 tcpdump
1.2 Install Boost Headers
For Debian/Ubuntu systems
$ sudo apt install libboost-iostreams-dev libboost-random-dev
For RedHat/CentOS systems
$ sudo yum install boost-devel
2. Install Hyperscan
Now the latest release version is 5.4.1.
2.1 Install 5.4.1
To download and extract
$ wget https://github.com/intel/hyperscan/archive/refs/tags/v5.4.1.tar.gz
$ tar -xvf v5.4.1.tar.gz
$ cd v5.4.1
To build and install
$ mkdir build
$ cd build
$ cmake -DBUILD_SHARED_LIBS=on -DCMAKE_BUILD_TYPE=Release ..
However, compiled errors may occur as
/hyperscan-5.4.1/src/nfa/nfa_dump_dispatch.cpp:78:23: error: ‘LBR_NFA_VSHUF’ was not declared in this scope; did you mean ‘LBR_NFA_SHUF’?
78 | DISPATCH_CASE(LBR_NFA_VSHUF, LbrVShuf, dbnt_func);
/hyperscan-5.4.1/src/nfa/nfa_dump_dispatch.cpp:54:10: note: in definition of macro ‘DISPATCH_CASE’
54 | case dc_ltype: \/hyperscan-5.4.1/src/nfa/nfa_dump_dispatch.cpp:94:5: note: in expansion of macro ‘DISPATCH_BY_NFA_TYPE’
94 | DISPATCH_BY_NFA_TYPE(_dump(nfa, base));/hyperscan-5.4.1/src/nfa/nfa_dump_dispatch.cpp:55:9: error: ‘nfaExecLbrVShuf_dump’ was not declared in this scope; did you mean ‘nfaExecLbrShuf_dump’?
55 | nfaExec##dc_ftype##dc_func_call; \/hyperscan-5.4.1/src/nfa/nfa_dump_dispatch.cpp:78:9: note: in expansion of macro ‘DISPATCH_CASE’
78 | DISPATCH_CASE(LBR_NFA_VSHUF, LbrVShuf, dbnt_func);
/hyperscan-5.4.1/src/nfa/nfa_dump_dispatch.cpp:94:5: note: in expansion of macro ‘DISPATCH_BY_NFA_TYPE’
94 | DISPATCH_BY_NFA_TYPE(_dump(nfa, base));[ 72%] Building CXX object CMakeFiles/hs_compile_shared.dir/src/smallwrite/smallwrite_dump.cpp.o
make[2]: *** [CMakeFiles/hs_compile_shared.dir/build.make:2296: CMakeFiles/hs_compile_shared.dir/src/nfa/nfa_dump_dispatch.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs….
make[1]: *** [CMakeFiles/Makefile2:393: CMakeFiles/hs_compile_shared.dir/all] Error 2
make: *** [Makefile:130: all] Error 2
There already has been a pull 391 to fix the LBR_NFA_VSHUF error, but the pull request hasn’t merged into the release yet. The pull can fix the compilation error.
2.2 Install 5.4.0
Alternatively, it can also install the 5.4.0 version. Following similar commands to configure and compile, also errors occurred during compilation:
[ 92%] Linking CXX executable ../bin/patbench
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference toavx2_memset'core2_memcpy’
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference to
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference tocore2_memset'core2_memmove’
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference to
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference toavx2_memmove'corei7_memset’
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference to
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference tocorei7_snprintf'core2_snprintf’
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference to
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference toavx2_snprintf'corei7_memcpy’
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference to
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference toavx2_memcpy'corei7_memmove’
/usr/bin/ld: ../lib/libhs.so.5.4.0: undefined reference to
collect2: error: ld returned 1 exit status
make[2]: *** [examples/CMakeFiles/patbench.dir/build.make:104: bin/patbench] Error 1
make[1]: *** [CMakeFiles/Makefile2:804: examples/CMakeFiles/patbench.dir/all] Error 2…
Following the instructions of issue 292 can fix the issue.
After compile and install successfully, it can locate the install libraries in /usr/local/:
~ find /usr/local/ -name "libhs*"
/usr/local/lib/pkgconfig/libhs.pc
/usr/local/lib/libhs.so.5.4.1
/usr/local/lib/libhs.so.5.4.0
/usr/local/lib/libhs_runtime.so
/usr/local/lib/libhs_runtime.so.5.4.0
/usr/local/lib/libhs_runtime.so.5.4.1
/usr/local/lib/libhs_runtime.so.5
/usr/local/lib/libhs.so
/usr/local/lib/libhs.so.5
To compile your program with the installed library, simply put following flags during compilation:
-I<path to the hyperscan src directory> -L/usr/local/lib/ -lhs
If installed sqlite and pcre libraries, provided tools such as hscheck and hsbench and so on will also be compiled during the build process, located under build/bin
$ pwd
~/hyperscan-5.4.0/build/bin
➜ bin ll
total 2.9M
-rwxrwxr-x 1 zeek zeek 86K Mar 20 23:28 hscheck
-rwxrwxr-x 1 zeek zeek 57K Mar 20 23:28 patbench
-rwxrwxr-x 1 zeek zeek 47K Mar 20 23:28 pcapscan
-rwxrwxr-x 1 zeek zeek 18K Mar 20 23:28 simplegrep
-rwxrwxr-x 1 zeek zeek 2.7M Mar 20 23:28 unit-hyperscan