[TOC]
cat table2.txt | perl -p -e 's/\s+$/,/g'
replace all whitespace and newlines to comma
sed replace and remove
sed s stands for "substitute" - replacing one piece of text with another
sed 's/old-text/new-text/g' source.txt
(this will display your intended result in the console)
sed 's/old-text/new-text/g' source.txt > newfile.txt
(this will save your intended changes to a new file)
sed -i 's/old-text/new-text/g' source.txt
(CAREFUL, modifies the file with the changes)
sed -i '/username@someserver.com/d' /home/ubuntu/.bash_history
CAREFUL, -i will actually execute and delete all lines which contain username@someserver
BASIC SUBSTITION EXAMPLES
sed 's/old-text//g' source.txt
removes the old-text
sed 's/old-text/ /g' source.txt
replaces the old-text with a single space
sed 's/firstword//g;s/secondword//g' source.txt
deletes two words in one command
MORE COMPLICATED SUBSTITION
\n = newline
\t = tab
^ = beginning of a line
$ = end of a line
. = single character match
* = zero or more occurences of the previous character
[] = matches all of the characters in the brackets
ESCAPE CHARACTER ‘*’ matches a single asterisk rather than zero or more backslashes
sed 's/ *//g' source.txt
remove all whitespace
sed 's:.*/::' source.txt
remove all characters before and including the last slash
sed 's/ /\t /g' source.txt
replaces every space with a tab character instead
sed -e 's/$/\r/' source.txt
adds a windows compatible carriage return at the end of each line
sed 's/^#\(.*\)partner/\1partner/g' test.txt
captures the group of text between # and partner, used as variable \1
sed 'N;s/\n / /;P;D;'
removes the newline if the second line begins with a space
- N = add the next line to the work buffer
- P = print the top line of the work buffer
- D = delete the top line from the work buffer and run the script again
TO ESCAPE A SINGLE QUOTE
sed 's/'\''vid/'\''test/g'
replaces 'vid' with 'test'
sed 's/vid'\'' \=> '\''4/test'\'' \=< '\'5'/g'
replaces vid' => '4 with test' =< '5
To escape a unicode character
Sometimes if you open a document that was written in windows or with real unicode it will not be displayed properly.
In order to replace those characters from the command line try:
sed 's/\x85/.../g' some-file.md
sed -i 's/\x85/.../g' *.md
the first command just shows the expected output of the x85 (UTF-8 horizontal ellipsis) being replaced with three dots the second substitutes all of those UTF-8 characters for three dots in all .md (markdown) files in the current directory
when opened with geany text editor it at least shows:
- PAD = x80 = padding character (which was really just a dash)
- NEL = x85 = next line (which was really just a single character horizontal ellipsis)
- PU1 = x91 = private use 1 (which was just an apostrophe)
- PU2 = x92 = private use 2 (which was just an apostrophe)
- STS = x93 = set transmit state (which was just begin double quotes)
- CCH = x94 = cancel character (which was just end double quotes)
- SPA = x96 = start of protected error (which was just a long dash)
- EPA = x97 = end of protected error (which was just a long dash)
sed replace without altering the modified timestamp
!/bin/bash
for every file in the directory create a temp file and replace the invalid characters
SOURCE=./research/*
for FILE in $SOURCE; do
echo $FILE
TMP=`mktemp /tmp/file-XXXXXX`;
cp -p $FILE $TMP;
sed -i 's/\x85/.../g' $TMP;
sed -i "s/\x92/'/g" $TMP;
sed -i 's/\x93/"/g' $TMP;
sed -i 's/\x94/"/g' $TMP;
sed -i 's/\x96/-/g' $TMP;
sed -i 's/\x97/-/g' $TMP;
touch -r $FILE $TMP;
cp -p $TMP $FILE;
rm -f $TMP;
done
BASIC OTHER FUNCTIONS (-e = evaulate expression)
sed -e '1d' source.txt | more
delete one line and pipe the result to more
sed -e '2,6d' source.txt | head
delete lines 2 through 6 and pipe the result to head
sed -e '/^#/d' source.txt | more
delete lines which begin with a hash symbol
sed 's/^#\(.*\)//g' squid.conf > test.txt
replace a line beginning with a hash symbol with a blank line
sed '/^$/d' test.txt
delete blank lines
sed 's/^#.*//g' squid.conf | sed '/^$/d'
s/^\(.\{81\}\).*$/\1/ (delete the 81'st line?)
http://www.ibm.com/developerworks/linux/library/l-sed1.html
sed /s/'texttoreplace'/'replacementtext'/g/ filename > filename.txt
BASH SCRIPTING ALLOWS VARIABLES
sed 's/'$1'/new-text/g' $SOURCE.TXT
replace the first parameter passed in with new-text