[TOC]

cat table2.txt | perl -p -e 's/\s+$/,/g'

replace all whitespace and newlines to comma

sed replace and remove

sed s stands for "substitute" - replacing one piece of text with another

sed 's/old-text/new-text/g' source.txt

(this will display your intended result in the console)

sed 's/old-text/new-text/g' source.txt > newfile.txt

(this will save your intended changes to a new file)

sed -i 's/old-text/new-text/g' source.txt

(CAREFUL, modifies the file with the changes)

sed  -i '/username@someserver.com/d' /home/ubuntu/.bash_history

CAREFUL, -i will actually execute and delete all lines which contain username@someserver

BASIC SUBSTITION EXAMPLES

sed 's/old-text//g' source.txt

removes the old-text

sed 's/old-text/ /g' source.txt

replaces the old-text with a single space

sed 's/firstword//g;s/secondword//g' source.txt

deletes two words in one command

MORE COMPLICATED SUBSTITION

\n = newline
\t = tab
^ = beginning of a line
$ = end of a line
. = single character match
* = zero or more occurences of the previous character
[] = matches all of the characters in the brackets

ESCAPE CHARACTER ‘*’ matches a single asterisk rather than zero or more backslashes

sed 's/ *//g' source.txt

remove all whitespace

sed 's:.*/::' source.txt

remove all characters before and including the last slash

sed 's/ /\t /g' source.txt

replaces every space with a tab character instead

sed -e 's/$/\r/' source.txt

adds a windows compatible carriage return at the end of each line

sed 's/^#\(.*\)partner/\1partner/g' test.txt

captures the group of text between # and partner, used as variable \1

sed 'N;s/\n / /;P;D;'

removes the newline if the second line begins with a space

N = add the next line to the work buffer
P = print the top line of the work buffer
D = delete the top line from the work buffer and run the script again

TO ESCAPE A SINGLE QUOTE

sed 's/'\''vid/'\''test/g'

replaces 'vid' with 'test'

sed 's/vid'\'' \=> '\''4/test'\'' \=< '\'5'/g'

replaces vid' => '4 with test' =< '5

To escape a unicode character

Sometimes if you open a document that was written in windows or with real unicode it will not be displayed properly.

In order to replace those characters from the command line try:

sed 's/\x85/.../g' some-file.md
sed -i 's/\x85/.../g' *.md

the first command just shows the expected output of the x85 (UTF-8 horizontal ellipsis) being replaced with three dots the second substitutes all of those UTF-8 characters for three dots in all .md (markdown) files in the current directory

when opened with geany text editor it at least shows:

PAD = x80 = padding character (which was really just a dash)
NEL = x85 = next line (which was really just a single character horizontal ellipsis)
PU1 = x91 = private use 1 (which was just an apostrophe)
PU2 = x92 = private use 2 (which was just an apostrophe)
STS = x93 = set transmit state (which was just begin double quotes)
CCH = x94 = cancel character (which was just end double quotes)
SPA = x96 = start of protected error (which was just a long dash)
EPA = x97 = end of protected error (which was just a long dash)

sed replace without altering the modified timestamp

!/bin/bash

for every file in the directory create a temp file and replace the invalid characters

SOURCE=./research/*
for FILE in $SOURCE; do
  echo $FILE
  TMP=`mktemp /tmp/file-XXXXXX`;
  cp -p $FILE $TMP;
  sed -i 's/\x85/.../g' $TMP;
  sed -i "s/\x92/'/g" $TMP;
  sed -i 's/\x93/"/g' $TMP;
  sed -i 's/\x94/"/g' $TMP;
  sed -i 's/\x96/-/g' $TMP;
  sed -i 's/\x97/-/g' $TMP;
  touch -r $FILE $TMP;
  cp -p $TMP $FILE;
  rm -f $TMP;
done

BASIC OTHER FUNCTIONS (-e = evaulate expression)

sed -e '1d' source.txt | more

delete one line and pipe the result to more

sed -e '2,6d' source.txt | head

delete lines 2 through 6 and pipe the result to head

sed -e '/^#/d' source.txt | more

delete lines which begin with a hash symbol

sed 's/^#\(.*\)//g' squid.conf > test.txt

replace a line beginning with a hash symbol with a blank line

sed '/^$/d' test.txt

delete blank lines

sed 's/^#.*//g' squid.conf | sed '/^$/d'


s/^\(.\{81\}\).*$/\1/  (delete the 81'st line?)

http://www.ibm.com/developerworks/linux/library/l-sed1.html

sed /s/'texttoreplace'/'replacementtext'/g/ filename > filename.txt

BASH SCRIPTING ALLOWS VARIABLES

sed 's/'$1'/new-text/g' $SOURCE.TXT

replace the first parameter passed in with new-text

sed substitute replace text or a line or remove a line or utf-8