grep


A simple way to find which DHCP server gave you an IP

Recently, we were trying to find which DHCP server was responding to the messages on the network. Using a DHCP-enabled client on a Fedora 26 GNU/Linux we grepped the contents of journalctl to find the DHCP acknowledgment messages (DHCPACK) and figure out the IP of the DHCP server.

The command we used was the following:

sudo journalctl | grep DHCPACK;

And it gave us results such as the ones below:

[[email protected] ~]$ sudo journalctl | grep DHCPACK
Nov 12 13:08:28 sys-net dhclient[578]: DHCPACK from 10.1.101.252 (xid=0x80ec760c)
Nov 12 13:08:34 sys-net dhclient[720]: DHCPACK from 10.1.101.252 (xid=0x2ed6486f)
Nov 12 11:51:19 sys-net dhclient[1248]: DHCPACK from 10.1.101.252 (xid=0xe3dd491c)
Nov 12 12:02:09 sys-net dhclient[1407]: DHCPACK from 10.1.101.252 (xid=0x1fa42c2d)
Nov 12 12:11:03 sys-net dhclient[1508]: DHCPACK from 10.1.101.252 (xid=0x91c3990a)
Nov 12 12:14:06 sys-net dhclient[1607]: DHCPACK from 10.1.101.252 (xid=0x57ebb515)
Nov 12 12:19:27 sys-net dhclient[1710]: DHCPACK from 10.1.101.252 (xid=0x5450c250)
Nov 12 12:19:39 sys-net dhclient[1776]: DHCPACK from 10.1.101.252 (xid=0x2c38d517)
Nov 12 12:39:53 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:40:51 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:41:51 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:42:44 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:43:33 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:44:31 sys-net dhclient[1837]: DHCPACK from 192.168.1.1 (xid=0xe7a1182d)
Nov 12 12:46:20 sys-net dhclient[2053]: DHCPACK from 192.168.1.1 (xid=0xbb006001)

It is important to use sudo or else you will not be seeing messages from other users and the system. As, only users in groups ‘adm’, ‘systemd-journal’, ‘wheel’ can see all messages.

Advertisements

How to find lines that contain only lowercase characters

To print all lines that contain only lower case characters, we used the following regular expression in grep:

egrep '^[[:lower:]]+$' <file>;
#If you do not have egrep, use
grep -e '^[[:lower:]]+$' <file>;

Breakdown of the above regular expression:

  • ^ instructs the regular expression parser that the pattern should always start with the beginning of the line
  • [[:lower:]] this special instruction informs us that only lower case characters can match it
  • + the plus sign causes the preceding token to be matched one or more times
  • $ signifies the end of the line

Ignore all edits to a file that is committed in git

Recently, we were working on a project that had committed in the source code a configuration file. That configuration file had hard-coded the production system values, so we had to modify them to the development system values before using it.

To avoid committing the configuration file with the development parameters by accident, we instructed git to ignore any changes that were made to it using the following command.

git update-index --assume-unchanged <file>;

By doing so, git assumed that the file was always unchanged and it never showed up in the git status results nor was staged when git add . was used etc.

After we were done with development (and whenever we needed to pull the branch for changes or checkout another branch) we removed the file from the list of ignored files using the following command.

git update-index --no-assume-unchanged <file>;

Using this command, git would start again to monitor changes to the file and merge it or update it or push it when needed as it would normally do for any file not included in the .gitignore file. The best part of this trick is that you do not have to update the .gitignore file to achieve the task of ignoring a file.

More information

git update-index modifies the index or directory cache. Each file mentioned is updated into the index and any unmerged or needs updating state is cleared.

--[no-]assume-unchanged When these flags are specified, the object names recorded for the paths are not updated. Instead, these options set and unset the “assume unchanged” bit for the paths. When the “assume unchanged” bit is on, Git stops checking the working tree files for possible modifications, so you need to manually unset the bit to tell Git when you change the working tree file. This is sometimes helpful when working with a big project on a filesystem that has very slow lstat(2) system call (e.g. cifs).

This option can be also used as a coarse file-level mechanism to ignore uncommitted changes in tracked files (akin to what .gitignore does for untracked files). Git will fail (gracefully) in case it needs to modify this file in the index e.g. when merging in a commit; thus, in case the assumed-untracked file is changed upstream, you will need to handle the situation manually.

From: man git-update-index

Bonus

In case you are wondering on how to see which files are currently ignored in your local repository copy by the git update-index --assume-unchanged <file>; command, you can use the following code:

git ls-files -v | grep -e '^[[:lower:]]';

git ls-files -v will print out all objects that git knows and the -v parameter will print all flags associated with them. The files that are ignored because of the
git update-index --assume-unchanged <file>; command will be printed each one on a different line that starts with a lower case character. So, to get all files that are ignored by the git update-index --assume-unchanged <file>; command, we need to grep the results of git ls-files -v for lines that start with a lower case.

git-ls-files shows information about files in the index and the working tree.
It merges the file listing in the directory cache index with the actual working directory list, and shows different combinations of the two.

-v Similar to -t (below), but use lowercase letters for files that are marked as assume unchanged (see git-update-index(1)).

-t This feature is semi-deprecated. For scripting purpose, git-status(1)–porcelain and git-diff-files(1)–name-status are almost always superior alternatives, and users should look at git-status(1)–short or git-diff(1)–name-status for more user-friendly alternatives.
This option identifies the file status with the following tags (followed by a space) at the start of each line:

An additional interesting parameter for git ls-files is
-i, --ignored Shows only ignored files in the output. When showing files in the index, it prints only those matched by an exclude pattern. When showing “other” files, it shows only those matched by an exclude pattern.

--exclude-standard Add the standard Git exclusions: .git/info/exclude, .gitignore in each directory, and the user’s global exclusion file.
From: man git-ls-files

Examples for git ls-files -i, –ignored and –exclude-standard

# Show files in the index that are ignored because of patterns in .gitignore
git ls-files --ignored --exclude-from=.gitignore;
# Show other (i.e. untracked) files that are ignored because of patterns in .gitignore
git ls-files --ignored --other --exclude-from=.gitignore;
# Show files in the index that are ignored because of patterns in any of the standard git exclusions.
git ls-files --ignored --exclude-standard;
# Show other (i.e. untracked) files that are ignored because of patterns in any of the standard git exclusions.
git ls-files --ignored --exclude-standard --other;


grep: How to match lines using any of multiple patterns

Recently, we needed to filter the results of ps x using two different patterns.
The first pattern was ./ where we needed to match that exact character sequence.
The . period character is treated as a special character in regular expressions (it matches a single character of any value, except for the end of line), so we decided to use the -F parameter to remove this special handling.
Doing this change prevented us from writing a regular expression that uses the OR | operator.

-F (or --fixed-strings) is a matching control option that instructs grep to interpret the patterns as a list of fixed strings (instead of regular expressions), separated by newlines, any of which is to be matched.
We tried assigning the different patterns as different lines to a variable and then using them on the pipe, like in the following example:

patterns="./
banana";
ps x | grep -F $patterns;

..but it failed.

Solution

grep supports a matching control option -e that allows us to define multiple patterns using different strings.

-e PATTERN (or --regexp=PATTERN) uses the value PATTERN as the pattern. If this option is used  multiple times or it is combined with the -f (--file) option, grep will search for all patterns given.

In the end, our command was transformed to the following, which worked just fine!

ps x | grep -F -e "./" -e "banana";

How to search for specific filenames in .tar archives

The following commands will search in the .tar archives found in the specified folder and print on screen all files that their paths or filenames match our search token. We provide multiple solutions, each one for a different type of .tar archive depending on the compression used.

For .tar archives

find /media/repository/packages/ -type f -iname "*.tar" -exec tar -t -f '{}' \; | grep "configurations/arm-cortexa9";

For .tar.bz2 archives

find /media/repository/packages/ -type f -iname "*.tar.bz2" -exec tar -t -j -f '{}' \; | grep "configurations/arm-cortexa9";

For .tar.xz archives

find /media/repository/packages/ -type f -iname "*.tar.xz" -exec tar -t -J -f '{}' \; | grep "configurations/arm-cortexa9";

For .tar.gz and .tgz archives

Please note that this commands uses the -o (which is the logical or) parameter on find to search for multiple filename extensions.

find /media/repository/packages/ -type f \( -iname "*.tar.gz" -o -iname "*.tgz" \) -exec tar -t -z -f '{}' \; | grep "configurations/arm-cortexa9";

find Parameters Legend

  • -type f filters out any result which is not a regular file
  • -exec command '{}' \; runs the specified command on the results of find. The string '{}' is replaced by the current file name being processed.
  • -o is the logical Or operator. The second expression  is not evaluated if the first expression is true.

tar Parameters Legend

  • -z or --gzip instructs tar to filter the archive through gzip
  • -j or --bzip2 filters the archive through bzip2
  • -J or --xz filters the archive through xz
  • -t or --list lists the contents of an archive
  • -f or --file=INPUT uses the archive file or device named INPUT