Tuesday, April 19, 2005

Use ls -h to show Kilobyte, Megabyte, Gigabyte, etc

Use the -h switch when doing an ls to get a more readable filesize value (K for kilobyte,

M for megabyte, G for gigabyte), e.g.:

[root@server backup]# ls -altrh
total 9.4G
drwxr-xr-x 2 4294967294 4294967294 4.0K Oct 6 2004 last.txt
-rw-r--r-- 1 4294967294 4294967294 4.3G Oct 6 2004 backup.tgz
-rw-r--r-- 1 4294967294 4294967294 843M Nov 23 14:31 backup_20041123.tgz
-rw-r--r-- 1 4294967294 4294967294 863M Dec 4 02:33 backup_20041204.tgz
-rw-r--r-- 1 4294967294 4294967294 5.9M Jan 17 01:55 20050117.tgz
-rw-r--r-- 1 4294967294 4294967294 928M Jan 17 02:49 backup_20050117.tgz
-rw-r--r-- 1 4294967294 4294967294 960M Feb 2 15:03 backup_20050202.tgz
drwxr-xr-x 23 root root 4.0K Feb 25 18:12 ..
drwxrwxrwx 3 root root 4.0K Apr 3 01:28 .
-rw-r--r-- 1 4294967294 4294967294 1.6G Apr 3 03:36 backup_20050403.tgz
[root@server backup]#

Sunday, April 17, 2005

curl Examples

curl http://curl.haxx.se

GET form:
curl "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK"

POST form:
curl -d "birthyear=1905&press=%20OK%20" www.hotmail.com/when/junk.cgi

File Upload POST:
curl -F upload=@localfilename -F press=OK [URL]

PUT:
curl -T uploadfile www.uploadhttp.com/receive.cgi



The site might require a different authentication method (check the headers
returned by the server), and then --ntlm, --digest, --negotiate or even
--anyauth might be options that suit you.
Sometimes your HTTP access is only available through the use of a HTTP
proxy. This seems to be especially common at various companies. A HTTP proxy
may require its own user and password to allow the client to get through to
the Internet. To specify those with curl, run something like:

curl -U proxyuser:proxypassword curl.haxx.se

If your proxy requires the authentication to be done using the NTLM method,
use --proxy-ntlm, if it requires Digest use --proxy-digest.

If you use any one these user+password options but leave out the password
part, curl will prompt for the password interactively.


Use curl to set the referer field with:

curl -e http://curl.haxx.se daniel.haxx.se


User Agent:
To make curl look like Internet Explorer on a Windows 2000 box:

curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]

Or why not look like you're using Netscape 4.73 on a Linux (PIII) box:

curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]


Redirects:
To tell curl to follow a Location:
curl -L www.sitethatredirects.com

If you use curl to POST to a site that immediately redirects you to another
page, you can safely use -L and -d/-F together. Curl will only use POST in
the first request, and then revert to GET in the following operations.



Cookies:
The simplest way to send a few cookies to the server when getting a page with
curl is to add them on the command line like:

curl -b "name=Daniel" www.cookiesite.com

Curl has a full blown cookie parsing engine built-in that comes to use if you
want to reconnect to a server and use cookies that were stored from a
previous connection (or handicrafted manually to fool the server into
believing you had a previous connection). To use previously stored cookies,
you run curl like:

curl -b stored_cookies_in_file www.cookiesite.com

Curl's "cookie engine" gets enabled when you use the -b option. If you only
want curl to understand received cookies, use -b with a file that doesn't
exist. Example, if you want to let curl understand cookies from a page and
follow a location (and thus possibly send back cookies it received), you can
invoke it like:

curl -b nada -L www.cookiesite.com

Curl has the ability to read and write cookie files that use the same file
format that Netscape and Mozilla do. It is a convenient way to share cookies
between browsers and automatic scripts. The -b switch automatically detects
if a given file is such a cookie file and parses it, and by using the
-c/--cookie-jar option you'll make curl write a new cookie file at the end of
an operation:

curl -b cookies.txt -c newcookies.txt www.cookiesite.com

HTTPS:
Curl supports encrypted fetches thanks to the freely available OpenSSL
libraries. To get a page from a HTTPS server, simply run curl like:

curl https://that.secure.server.com

Certificates:
Use a certificate with curl on a HTTPS server
like:

curl -E mycert.pem https://that.secure.server.com

Custom Request Elements

Doing fancy stuff, you may need to add or change elements of a single curl
request.

For example, you can change the POST request to a PROPFIND and send the data
as "Content-Type: text/xml" (instead of the default Content-Type) like this:

curl -d "" -H "Content-Type: text/xml" -X PROPFIND url.com

You can delete a default header by providing one without content. Like you
can ruin the request by chopping off the Host: header:

curl -H "Host:" http://mysite.com

You can add headers the same way. Your server may want a "Destination:"
header, and you can add it:

curl -H "Destination: http://moo.com/nowhere" http://url.com


Debug

Many times when you run curl on a site, you'll notice that the site doesn't
seem to respond the same way to your curl requests as it does to your
browser's.

Then you need to start making your curl requests more similar to your
browser's requests:

* Use the --trace-ascii option to store fully detailed logs of the requests
for easier analyzing and better understanding

* Make sure you check for and use cookies when needed (both reading with -b
and writing with -c)

* Set user-agent to one like a recent popular browser does

* Set referer like it is set by the browser

* If you use POST, make sure you send all the fields and in the same order as
the browser does it. (See chapter 4.5 above)

A very good helper to make sure you do this right, is the LiveHTTPHeader tool
that lets you view all headers you send and receive with Mozilla/Firefox
(even when using HTTPS).

Find number of occurrences of a word

To get the number of occurrences of a word (e.g. ServerName) in a file (e.g. httpd.conf), do a:

grep -c ServerName httpd.conf

For a line occurrence frequency count report:

sort httpd.conf | uniq -c | sort -nr

For a word occurrence frequency count report:

sed -e 's/\.//g' -e 's/ //g' "httpd.conf" | tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr

ulimit example

[root@server conf]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 4095
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[root@server conf]# ulimit -a | grep files
open files (-n) 1024

Change (example of increasing to 5000 files):

ulimit -n 5000

Rename file starting with a dash(-)

If having problem renaming a file that starts with a dash:

Since the file name begins with a '-' it looks like an option to the
command. You need to force it to not look like an option. Put a ./
in the front of it. Or give it the full file name path. Or tell the
command you are through with options by using the double dash to end
all option processing. This is common to most traditional UNIX
commands.

cp ./* /some/folder
cp -- * /some/folder

And the same for other utilities too.

mv ./-stuff differentstuff
mv -- -stuff differentstuff

Resolve "device is busy" error when doing a umount

If get "device is busy" when doing a umount, e.g.:

[root@server/]# umount /home/data/
umount: /home/data: device is busy


To find out who's still using the share, do a, e.g.:

[root@orion /]# fuser -u -v /home/data/

USER PID ACCESS COMMAND
/home/data/ root 15376 ..c.. vim

Delete a range of lines in Vim example

In vim, to delete a range of lines, e.g. lines 10 through line 290, do a:

:10,290d

Have Vim remeber last cursor position

to have vim remeber the last cursor position of file edit, put in .vimrc:

set viminfo='10,\"100,:20,%,n~/.viminfo
au BufReadPost * if line("'\"") > 0|if line("'\"") <= line("$")|exe("norm '\"")|else|exe "norm $"|endif|endif

Enabling syntax highlighting in Vim

add:

syntax enable

to .vimrc to enable syntax highlighting in vim (e.g. when editing php or html files)

Prevent others from browsing your files

In a multiuser server environment, set diretories to these settings to prevent others from browsing your files: (read/write/execute by owner, & only execute by others, no read/write/execute by group)

e.g.
Read Write eXecute

Owner Access On On On
Group Access Off Off Off
Other Access Off Off On

chmod 711 /usr/home/USERNAME/
chmod 701 /usr/www/users/USERNAME/

Show tabs & end of lines in vim

Do a:

:set list

Redhat/ Fedora linux set services automatically start

e.g. to set nfs to start automatically:

[root@www2 rc3.d]# chkconfig --list nfs
nfs 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@www2 rc3.d]# chkconfig nfs on
[root@www2 rc3.d]# chkconfig --list nfs
nfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off


alternatively, e.g. for mysql:

# To make the script be run on machine start up we need to create a symbolic link to it.

cd /etc/rc.d/rc3.d
ln -s ../init.d/mysql S85mysql
cd /etc/rc.d/rc5.d
ln -s ../init.d/mysql S85mysql

# Now the links to make MySQL shut down:

cd /etc/rc.d/rc0.d
ln -s ../init.d/mysql K85mysql
cd /etc/rc.d/rc6.d
ln -s ../init.d/mysql K85mysql


Runlevel Links

Some services depend on other services. The 'httpd' service (Apache web server) for example won't start correctly if the 'network' script hasn't already set up the network interfaces. How is the order in which services are started on boot determined?

Have a look at the '/etc/rc.d' directory:

$ ls /etc/rc.d

init.d/ rc0.d/ rc2.d/ rc4.d/ rc6.d/ rc.local* rc.sysinit*
rc* rc1.d/ rc3.d/ rc5.d/ rc.firewall rc.modules*

You see the 'init.d' from '/etc' here again (in fact it's the same) and then several directories and files starting with 'rc' ('rc' is short for 'runcom[mand]').
In Mandrake Linux releases 8.0 and later, these files and directories are also accessible directly from the '/etc' directory.

If you now look into one of those 'rcnumber' subdirectories, you will find a bunch of files, some of them starting with 'S' and some of them with 'K' followed by a two-digit number. 'S' is short for 'start' and 'K' stands for 'kill'. The numbers imply the order in which starting and killing services takes place. In fact all those files are just links to their appropriate counterparts in '/etc/init.d'.
'S12syslog' for example is a link to '/etc/init.d/syslog' and gets started after 'S10network' which links to '/etc/init.d/internet' but before 'S20random'.



Also for mysql:


shell> cp mysql.server /etc/init.d/mysql
shell> chmod +x /etc/init.d/mysql

Older Red Hat systems use the `/etc/rc.d/init.d' directory rather than `/etc/init.d'. Adjust the preceding commands accordingly. Alternatively, first create `/etc/init.d' as a symbolic link that points to `/etc/rc.d/init.d':

shell> cd /etc
shell> ln -s rc.d/init.d .

After installing the script, the commands needed to activate it to run at system startup depend on your operating system. On Linux, you can use chkconfig:

shell> chkconfig --add mysql

On some Linux systems, the following command also seems to be necessary to fully enable the mysql script:

shell> chkconfig --level 345 mysql on

On FreeBSD, startup scripts generally should go in `/usr/local/etc/rc.d/'. The rc(8) manual page states that scripts in this directory are executed only if their basename matches the *.sh shell filename pattern. Any other files or directories present within the directory are silently ignored. In other words, on FreeBSD, you should install the `mysql.server' script as `/usr/local/etc/rc.d/mysql.server.sh' to enable automatic startup.

As an alternative to the preceding setup, some operating systems also use `/etc/rc.local' or `/etc/init.d/boot.local' to start additional services on startup. To start up MySQL using this method, you could append a command like the one following to the appropriate startup file:

/bin/sh -c 'cd /usr/local/mysql; ./bin/mysqld_safe --user=mysql &'

Fix Vim backspace doesn't work

Put:

:set bs=2

in .vimrc

Resolve "session setup failed: ERRDOS - ERRnoaccess (Access denied.)"

In samba, if get:

session setup failed: ERRDOS - ERRnoaccess (Access denied.)

while trying mount -a after editing fstab, e.g.:

//server/dir /home/dir smb username=administrator,password=123 0 0

then

add domain, e.g.:

//server/dir /home/dir smb username=administrator/domain_name,password=123 0 0

Vim bookmark position and returning to bookmark

ml Mark the current position with the bookmark named l
'l Move to the beginning of the line where mark l is.
`l Move to the character where the named mark l is.
`` Return to exact position of the previous mark
'' Return to the beginning of the line where the previous mark was.

Using nohup

nohup command &

Continue to execute command (which can include arguments) in the background after you logout. Technically, nohup starts command with the SIGHUP signal set to be ignored.
If standard output is the terminal then standard output is redirected to the file nohup.out. Output is appended to nohup.out if the file already exists. If standard error is the terminal then standard error is redirected to the same file as standard output.

Creating a Disk Array in Fedora

Create (mdadm --create) mode is used to create a new array. In this example I use mdadm to create a RAID-0 at /dev/md0 made up of /dev/sdb1 and /dev/sdc1:

# mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/sdb1 /dev/sdc1

mdadm: chunk size defaults to 64K

mdadm: array /dev/md0 started.
The --level option specifies which type of RAID to create in the same way that raidtools uses the raid-level configuration line. Valid choices are 0,1,4 and 5 for RAID-0, RAID-1, RAID-4, RAID-5 respectively. Linear (--level=linear) is also a valid choice for linear mode. The --raid-devices option works the same as the nr-raid-disks option when using /etc/raidtab and raidtools.


Use the --stop or -S command to stop running array:

# mdadm -S /dev/md0


mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/hdc1 /dev/hdd1


[root@localhost home]# mdadm --query /dev/md0
/dev/md0: 223.58GiB raid0 2 devices, 0 spares. Use mdadm --detail for more detail.
/dev/md0: No md super block found, not an md component.
[root@localhost home]# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.01
Creation Time : Thu Aug 19 14:49:32 2004
Raid Level : raid0
Array Size : 234436352 (223.58 GiB 240.06 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Thu Aug 19 14:49:32 2004
State : clean, no-errors
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Chunk Size : 64K

Number Major Minor RaidDevice State
0 22 1 0 active sync /dev/hdc1
1 22 65 1 active sync /dev/hdd1
UUID : 2f4c1a82:92bc9ece:372b0de6:c4cc65d6
Events : 0.2

Emacs sample commands (useful for default BASH shell and mysql console)

alt-f move forwards one word.
alt-b move backwards one word.
ctrl-e go to end of line.
ctrl-a go to beginning of line.

ctrl-k delete to end of line or delete blank line.
ctrl-_ undo last change

ESC-d delete word to the right


Readline vi Mode
While the Readline library does not have a full set of vi editing functions, it does contain enough to allow simple editing of the line. The Readline vi mode behaves as specified in the POSIX 1003.2 standard.

In order to switch interactively between emacs and vi editing modes, use the `set -o emacs' and `set -o vi' commands (see section 4.3 The Set Builtin). The Readline default is emacs mode.

Linux Hard drive benchmarking

Linux Hard drive benchmarking (on Dell 2400 2.2 GHz 7200 RPM 40 gig HDD)

[root@server shop]# hdparm -Tt /dev/hda

/dev/hda:
Timing buffer-cache reads: 1328 MB in 2.00 seconds = 664.00 MB/sec
Timing buffered disk reads: 124 MB in 3.00 seconds = 41.33 MB/sec

[root@server home]# hdparm /dev/hda

/dev/hda:
multcount = 16 (on)
IO_support = 0 (default 16-bit)
unmaskirq = 0 (off)
using_dma = 1 (on)
keepsettings = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 4863/255/63, sectors = 78125000, start = 0


[root@server home]# hdparm -u1 -m16 -c3 -d1 -X69 /dev/hda

/dev/hda:
setting 32-bit IO_support flag to 3
setting multcount to 16
setting unmaskirq to 1 (on)
setting using_dma to 1 (on)
setting xfermode to 69 (UltraDMA mode5)
multcount = 16 (on)
IO_support = 3 (32-bit w/sync)
unmaskirq = 1 (on)
using_dma = 1 (on)
[root@server home]# hdparm /dev/hda

/dev/hda:
multcount = 16 (on)
IO_support = 3 (32-bit w/sync)
unmaskirq = 1 (on)
using_dma = 1 (on)
keepsettings = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 4863/255/63, sectors = 78125000, start = 0
[root@server home]# hdparm -Tt /dev/hda
/dev/hda:
Timing buffer-cache reads: 1340 MB in 2.00 seconds = 670.00 MB/sec
Timing buffered disk reads: 142 MB in 3.03 seconds = 46.86 MB/sec

put command in /etc/rc.d/rc.local

How much swap space do you have?

The free command reports information on system-memory usage:

rutabaga% free
total used free shared buffers cached
Mem: 127888 126744 1144 27640 1884 51988
-/+ buffers: 72872 55016
Swap: 130748 23916 106832
All the numbers here are reported in 1024-byte blocks. Here, we see a system with 127,888 blocks (about 127 MB) of physical RAM, with 126,744 (about 126 MB) currently in use. Note that your system actually has more physical RAM than that given in the "total" column; this number does not include the memory used by the kernel for its own sundry needs.

The "shared" column lists the amount of physical memory shared between multiple processes. Here, we see that about 27 MB of pages are being shared, which means that memory is being utilized well. The "buffers" column shows the amount of memory being used by the kernel buffer cache. The buffer cache (described briefly in the previous section) is used to speed up disk operations, by allowing disk reads and writes to be serviced directly from memory. The buffer cache size will increase or decrease as memory usage on the system changes; this memory is reclaimed if it is needed by applications. Therefore, although we see that 126 MB of system memory is in use, not all (but most) of it is being used by application programs. The "cache" column indicates how many memory pages the kernel has cached for faster access later.

Since the memory used for buffers and cache can easily be reclaimed for use by applications, the second line (-/+ buffers/cache) provides an indication of the memory actually used by applications (the "used" column) or available to applications (the "free" column). The sum of the memory used by buffers and cache reported in the first line is subtracted from the total used memory and added to the total free memory to give the two figures on the second line.

In the third line, we see the total amount of swap, 130,748 blocks (about 128 MB). In this case, only very little of the swap is being used; there is plenty of physical RAM available. If additional applications were started, larger parts of the buffer cache memory would be used to host them. Swap space is generally used as a last resort when the system can't reclaim physical memory in other ways.

Note that the amount of swap reported by free is somewhat less than the total size of your swap partitions and files. This is because several blocks of each swap area must be used to store a map of how each page in the swap area is being utilized. This overhead should be rather small; only a few kilobytes per swap area.