Wednesday, September 9, 2009

Web Scalability and Performance

Web Scalability and Performance – Real Life Lessons

Last Saturday, we had TechWeekend #3 in Pune, on the theme of Website Scalability and Performance. Mukul Kumar, co-founder, and VP of Engineering at Pubmatic, talked about the hard lessons inscalability they learnt on their way to building a web service that serves billions of ad impressions per month.

Here are the slides used by Mukul. If you cannot see the slides, click here.
Web Scalability & Performance

The talk was live-tweeted by @punetechlive and @d7ylive. Here are a few highlights from the talk:

  • Keep it simple: If you cannot explain your application to your sales staff, you probably won’t be able to scale it!
  • Use JMeter to monitor performance, to a good job of scaling your site
  • Performance testing idea: Take 15/20 Amazon EC2 servers, run JMeter with 200threads on each for 10 hours. Bang on your website! (a few days later, @d7y pointed out that using openSTA instead of JMeter can give you upto 500 threads per server even on old machines.)
  • Scaling your application: have a loosely coupled, shared nothing, stateless, distributed architecture
  • Mysql scalability tip: Be careful before using new features, or new versions. Or don’t use them at all!
  • Website scalability: think global. Some servers in California, some servers in London, etc. Similarly, think global when designing your app. Having servers across the world will drive architecture decisions. When half your data-center is 3000 miles from the other half, interesting, non-trivial problems start cropping up. Also, think carefully about horizontal scaling (lots of cheap servers) vs vertical scaling (few big fat servers)
  • memcache tip: pre-populate memcache with most common objects
  • Scalability tip: Get a hardware load balancer (if you can afford one). Amazon AWS has some load-balancers, but they don’t perform so well
  • Remember the YouTube algo for scaling:
    while(1){
    identify_and_fix_bottlenecks();
    eat_drink();
    sleep();
    notice_new_bottleneck();
    }

    there’s no alternative to this.
  • Scalability tip: You can’t be sure of your performance unless you test with real load, real env, real hardware, real software!
  • Scalability tip – keep the various replicated copies of data loosely consistent. Speeds up your updates. But, figure out which parts of your database must be consistent at all times, and which ones can have “eventual consisteny”
  • Hard lessons: keep spare servers at all times. Keep servers independent – on failure shouldn’t affect other servers
  • Hard lessons: Keep all commands in a script. You will have to run them at 2am. Then 3am. Then 7am.
  • Hard lessons: Have a well defined process for fault identification, communication and resolution (because figuring these things out at 2am, with a site that is down, is terrible.)
  • Hard lessons: Monitor your web service from 12 cities around the world!
  • Hard lesson, Be Paranoid – At any time: servers can go down, DDOS can happen, NICs can become slow or fail!

Wednesday, May 27, 2009

canonical URL

Canonical URL:

Some websites differentiates some.com and www.some.com as two different websites, but actually it is not.

Example:

Justmeans.com
&
www.justmeans.com

Those are just the “same” ones.

Solution:
replace non www URL with www url and redirect it.

$strNewRedirection = preg_replace('/(^http:\/\/www\.|^www\.|^http:\/\/)/', '', $strServerName);
$strNewRedirection = "http://www.".$strNewRedirection.$_SERVER['REQUEST_URI'];
 header("Location: ".$strNewRedirection);
exit();

Saturday, May 9, 2009

Really a class analogy...

Really a class analogy...
An Old Story:

The Ant works hard in the withering heat all summer building its house and
laying up supplies for the winter.

The Grasshopper thinks the Ant is a fool and laughs & dances & plays the
summer away.

Come winter, the Ant is warm and well fed. The Grasshopper has no food or
shelter so he dies out in the cold.


Indian Version:


The Ant works hard in the withering heat all summer building its house and
laying up supplies for the winter.

The Grasshopper thinks the Ant's a fool and laughs & dances & plays the
summer away.

Come winter, the shivering Grasshopper calls a press conference and
demands to know why the Ant should be allowed to be warm and well fed
while others are cold and starving.
NDTV, BBC, CNN show up to provide pictures of the shivering Grasshopper
next to a video of the Ant in his comfortable home with a table filled
with food.

The World is stunned by the sharp contrast. How can this be that this poor
Grasshopper is allowed to suffer so?

Arundhati Roy stages a demonstration in front of the Ant's house.

Medha Patkar goes on a fast along with other Grasshoppers demanding that
Grasshoppers be relocated to warmer climates during winter .

Mayawati states this as `injustice' done on Minorities. 

Amnesty International and Koffi Annan criticize the Indian Government for
not upholding the fundamental rights of the Grasshopper.

The Internet is flooded with online petitions seeking support to the
Grasshopper (many promising Heaven and Everlasting Peace for prompt
support as against the wrath of God for non-compliance) .

Opposition MPs stage a walkout. Left parties call for 'Bengal Bandh' in
West Bengal and Kerala demanding a Judicial Enquiry.

CPM in Kerala immediately passes a law preventing Ants from working hard
in the heat so as to bring about equality of poverty among Ants and
Grasshoppers.

Lalu Prasad allocates one free coach to Grasshoppers on all Indian Railway
Trains, aptly named as the 'Grasshopper Rath'.

Finally, the Judicial Committee drafts the ' Prevention of Terrorism
Against Grasshoppers Act' [POTAGA], with effect from the beginning of the
winter.

Arjun Singh makes 'Special Reservation ' for Grasshoppers in Educational
Institutions & in Government Services. 


The Ant is fined for failing to comply with POTAGA and having nothing left
to pay his retroactive taxes,it's home is confiscated by the Government
and handed over to the Grasshopper in a ceremony covered by NDTV.



Arundhati Roy calls it ' A Triumph of Justice'.

Lalu calls it 'Socialistic Justice '. 

CPM calls it the ' Revolutionary Resurgence of the Downtrodden ' 

Koffi Annan invites the Grasshopper to address the UN General Assembly.




Many years later.... 

The Ant has since migrated to the US and set up a multi-billion dollar
company in Silicon Valley, 


100s of Grasshoppers still die of starvation despite reservation somewhere
in India, 


.
......AND




As a result of loosing lot of hard working Ants and feeding the
grasshoppers,

Thursday, April 16, 2009

Useful Linux Commands

Shorthand at the Command Prompt

Some of these are specific to the bash shell. I have not experimented enough with other shells to know which are common to all shells. See also the ``Bash Reference Card'', SSC (2000), available online.

  • / - root directory
  • ./ - current directory
  • ./command_name - run a command in the current directory when the current directory is not on the path
  • ../ - parent directory
  • ~ - home directory
  • $ - typical prompt when logged in as ordinary user
  • # - typical prompt when logged in as root or superuser
  • ! - repeat specified command
  • !! - repeat previous command
  • ^^ - repeat previous command with substitution
  • & - run a program in background mode
  • [Tab][Tab] - prints a list of all available commands. This is just an example of autocomplete with no restriction on the first letter.
  • x[Tab][Tab] - prints a list of all available completions for a command, where the beginning is ``x''
  • [Alt][Ctrl][F1] - switch to the first virtual text console
  • [Alt][Ctrl][Fn] - switch to the nth virtual text console. Typically, there are six on a Linux PC system.
  • [Alt][Ctrl][F7] - switch to the first GUI console, if there is one running. If the graphical console freezes, one can switch to a nongraphical console, kill the process that is giving problems, and switch back to the graphical console using this shortcut.
  • [ArrowUp] - scroll through the command history (in bash)
  • [Shift][PageUp] - scroll terminal output up. This also works at the login prompt, so you can scroll through your boot messages.
  • [Shift][PageDown] - scroll terminal output down
  • [Ctrl][Alt][+] - switch to next X server resolution (if the server is set up for more than one resolution)
  • [Ctrl][Alt][-] - change to previous X server resolution
  • [Ctrl][Alt][BkSpc] - kill the current X server. Used when normal exit is not possible.
  • [Ctrl][Alt][Del] - shut down the system and reboot
  • [Ctrl]c - kill the current process
  • [Ctrl]d - logout from the current terminal
  • [Ctrl]s - stop transfer to current terminal
  • [Ctrl]q - resume transfer to current terminal. This should be tried if the terminal stops responding.
  • [Ctrl]z - send current process to the background
  • reset - restore a terminal to its default settings
  • [Leftmousebutton] - Hold down left mouse button and drag to highlight text. Releasing the button copies the region to the text buffer under X and (if gpm is installed) in console mode.
  • [Middlemousebutton] - Copies text from the text buffer and inserts it at the cursor location. With a two-button mouse, click on both buttons simultaneously. It is necessary for three-button emulation to be enabled, either under gpm or in XF86Config.

Typical Dot Files

There is some redundancy across these programs. For example, the look and behavior of emacs can be customized by usinng the .emacs file, but also by adding the appropriate modifications to the .Xdefaults file. Default versions of these files are often installed in users' home directories when the software packages that use them are installed. If a program doesn't find its configuration file in the user's home directory, it will often fall back on a sytem-wide default configuration file installed in one of the subdirectories that the package lives in.

  • .bash_logout - file executed by bash shell on logout
  • .bash_profile - initialization of bash shell run only on login. Bash looks first for a .bash_profile file when started as a login shell or with the -login option. If it does not find .bash_profile, it looks for .bash_login. If it doesn't find that, it looks for .profile. System-wide functions and aliases go in /etc/bashrc and default environment variables go in /etc/profile.
  • .bashrc - initialization command run when bash shell starts up as a non-login shell
  • .cshrc - initialization commands that are run automatically (like autoexec.bat) when C shell is initiated
  • .emacs - configuration file for emacs editor
  • .fvwmrc - configuration file for fvwm window manager
  • .fvwm2rc - configuration file for fvwm2 window manager
  • .jedrc - configuration file for the jed text editor
  • .lessrc - typically contains key bindings for cursor movement with the less command
  • .login - initialization file when user logs in
  • .logout - commands run when user logs out
  • .wm_style - gives choice of default window manager if one is not specified in startx
  • .Xdefaults - sets up X resources for individual user. The behavior of many different application programs can be changed by modifying this file.
  • .xinitrc - initialization file when running startx. Can be used to activate applications, run a given window manager, and modify the appearance of the root window.
  • .xsession - configuration file for xdm

Useful Files

  • /boot/vmlinuz - the typical location and name of the Linux kernel. In the Slackware distribution, the kernel is located at /vmlinuz.
  • /dev/fd0 - first floppy disk drive
  • /dev/fd0H1440 - driver for the first floppy drive in high density mode. Generally, this is invoked when formatting a floppy drive for a particular density. Slackware comes with drivers that allow for formatting a 3.5" diskette with up to 1.7MB of space. Red Hat and Mandrake do not contain these device driver files by default.
  • /dev/fd1 - second floppy disk drive
  • /dev/hda - first IDE hard drive
  • /dev/hdc - on many machines, the IDE cdrom drive. Most often, there is a symbolic link called /dev/cdrom which is just a link to the true cdrom driver file.
  • /dev/null - used when you want to send output into oblivion
  • /etc/aliases - file containing aliases used by sendmail and other MTAs (mail transport agents). After updating this file, it is necessary to run the newaliases utility for the changes to be passed to sendmail.
  • /etc/bashrc - system-wide default functions and aliases for the bash shell
  • /etc/conf.modules - aliases and options for configurable modules
  • /etc/crontab - shell script to run different commands periodically (hourly, daily, weekly, monthly, etc.)
  • /etc/DIR_COLORS - used to store colors for different file types when using ls command. The dircolors command uses this file when there is not a .dir_colors file in the user's home directory. Used in conjunction with the eval command (see below).
  • /etc/exports - specifies hosts to which file systems can be exported using NFS. Man exports contains information on how to set up this file for remote users.
  • /etc/fstab - contains information on partitions and filesystems used by system to mount different partitions and devices on the directory tree
  • /etc/HOSTNAME - stores the name of the host computer
  • /etc/hosts - contains a list of host names and absolute IP addresses.
  • /etc/hosts.allow - hosts allowed (by the tcpd daemon) to access Internet services
  • /etc/hosts.deny - hosts forbidden (by the tcpd daemon) to access Internet services
  • /etc/group - similar to /etc/passwd but for groups
  • /etc/inetd.conf - configures the inetd daemon to tell it what TCP/IP services to provide (which daemons to load at boot time). A good start to securing a Linux box is to turn off these services unless they are necessary.
  • /etc/inittab - runs different programs and processes on startup. This is typically the program which is responsible for, among other things, setting the default runlevel, running the rc.sysinit script contained in /etc/rc.d, setting up virtual login terminals, bringing down the system in an orderly fashion in response to [Ctrl][Alt][Del], running the rc script in /etc/rc.d, and running xdm for a graphical login prompt (only if the default runlevel is set for a graphical login).
  • /etc/issue - pre-login message. This is often overwitten by the /etc/rc.d/rc.S script (in Slackware) or by the /etc/rc.d/rc.local script (in Mandrake and Red Hat, and perhaps other rpm-based distributions). The relevant lines should be commented out (or changed) in these scripts if a custom pre-login message is desired.
  • /etc/lilo.conf - configuration file for lilo boot loader
  • /etc/motd - message of the day file, printed immediately after login. This is often overwritten by /etc/rc.d/rc.S (Slackware) or /etc/rc.d/rc.local (Mandrake/Red Hat) on startup. See the remarks in connection with /etc/issue.
  • /etc/mtab - shows currently mounted devices and partitions and their status
  • /etc/passwd - contains passwords and other information concerning users who are registered to use the system. For obvious security reasons, this is readable only by root. It can be modified by root directly, but it is preferable to use a configuration utility such as passwd to make the changes. A corrupt /etc/passwd file can easily render a Linux box unusable.
  • /etc/printcap - shows the setup of printers
  • /etc/profile - sets system-wide defaults for bash shell. It is this file in Slackware that sets up the DIR_COLORS environment variable for the color ls command. Also sets up other system-wide environment variables.
  • /etc/resolv.conf - contains a list of domain name servers used by the local machine
  • /etc/securetty - contains a list of terminals on which root can login. For security reasons, this should not include dialup terminals.
  • /etc/termcap - ASCII database defining the capabilities and characteristics of different consoles, terminals, and printers
  • /etc/X11/XF86Config - X configuration file. The location in Slackware is /etc/XF86Config.
  • /proc/cpuinfo - cpu information
  • /proc/filesystems - prints filesystems currently in use
  • /proc/interrupts - prints interrupts currently in use
  • /proc/ioports - contains a list of the i/o addresses used by various devices connected to the computer
  • /proc/kcore - The command ls -l /proc/kcore will give the amount of RAM on the computer. It's also possible to use the free command to get the same information (and more).
  • /proc/version - prints Linux version and other info
  • /var/log/messages - used by syslog daemon to store kernel boot-time messages
  • /var/log/lastlog - used by system to store information about last boot
  • /var/log/wtmp - contains binary data indicating login times and duration for each user on system

Important Directories

Different distributions have different directory structures, despite attempts at standardization such as the the Linux Filesystem Hierarchy Standard (FHS) organization.

  • /bin - essential UNIX commands such as ls, etc. Should contain all binaries needed to boot the system or run it in single-user mode
  • /boot - files used during booting and possibly the kernel itself are stored here
  • /dev - contains device files for various devices on system
  • /etc - files used by subsystems such as networking, NFS, and mail. Includes tables of disks to mount, processes to run on startup, etc.
  • /etc/profile.d - contains scripts that are run by /etc/profile upon login.
  • /etc/rc.d - contains a number of shell scripts that are run on bootup at different run levels. There is also typically an rc.inet1 script to set up networking (in Slackwar), an rc.modules script to load modular device drivers, and an rc.local script that can be edited to run commands desired by the administrator, along the lines of autoexec.bat in DOS.
  • /etc/rc.d/init.d - contains most of the initialization scripts themselves on an rpm-based system.
  • /etc/rc.d/rc*.d - where ``*'' is a number corresponding to the default run level. Contains files for services to be started and stopped at that run level. On rpm-based systems, these files are symbolic links to the initialization scripts themselves, which are in /etc/rc.d/init.d.
  • /etc/skel - directory containing several example or skeleton initialization shells. Often contains subdirectories and files used to populate a new user's home directory.
  • /etc/X11 - configuration files for the X Window system
  • /home - home directories of individual users
  • /lib - standard shared library files
  • /lib/modules - modular device driver files, most with .o extensions
  • /mnt - typical mount point for many user-mountable devices such as floppy drives, cd-rom readers, etc. Each device is mounted on a subdirectory of /mnt.
  • /proc - virtual file system that provides a number of system statistics
  • /root - home directory for root
  • /sbin - location of binaries used for system administration, configuration, and monitoring
  • /tmp - directory specifically designed for programs and users to store temporary files.
  • /usr - directory containing a number of subdirectory with programs, libraries, documentation, etc.
  • /usr/bin - contains most user commands. Should not contain binaries necessary for booting the system, which go in /bin. The /bin directory is generally located on the same disk partition as /, which is mounted in read-only mode during the boot process. Other filesystems are only mounted at a later stage during startup, so putting binaries essential for boot here is not a good idea.
  • /usr/bin/X11 - most often a symbolic link to /usr/X11R6/bin, which contains executable binaries related to the X Window system
  • /usr/doc - location of miscellaneous documentation, and the main location of program documentation files under Slackware
  • /usr/include - standard location of include files used in C programs such as stdio.h
  • /usr/info - primary location of the GNU info system files
  • /usr/lib - standard library files such as libc.a. Searched by the linker when programs are compiled.
  • /usr/lib/X11 - X Window system distribution
  • /usr/local/bin - yet another place to look for comon executables
  • /usr/man - location of manual page files
  • /usr/sbin - other commands used by superuser for system administration
  • /usr/share - contains subdirectories where many installed programs have configuration, setup and auxiliary files
  • /usr/share/doc - location of program documentation files under Mandrake and Red Hat
  • /usr/src - location of source programs used to build system. Source code for programs of all types are often unpacked in this directory.
  • /usr/src/linux - often a symbolic link to a subdirectory whose name corresponds to the exact version of the Linux kernel that is running. Contains the kernel sources.
  • /var - administrative files such as log files, used by various utilities
  • /var/log/packages - contains files, each of which has detailed information on an installed package in Slackware. The same file can also be found at /var/adm/packages, since the adm subdirectory is a symbolic link to log. Each package file contains a short description plus a list of all installed files.
  • /var/log/scripts - package installation scripts in Slackware are stored here. You can inspect these scripts to see what special features are included in individual packages.
  • /var/spool - temporary storage for files being printed, mail that has not yet been picked up, etc.

Important Bash Shell Variables

These variables are most often inherited or declared when a shell is started. A great reference for bash shell variable, bash builtin commands, and bash in general is SSC (2000).

  • HOME - home directory, abbreviated as ~
  • MAIL - name of file that mail is stored in (mailbox)
  • MAILCHECK - sets the frequency at which bash checks for mail
  • PATH - directory paths to search for executable files. According to A Practical Guide to Linux, p.329, the PATH is set in /etc/profile. On my Linux box, /etc/profile adds /usr/X11R6/bin to the path, which means that the path must be set before by another configuration file. The .bash_profile file in my home directory adds /home/ambler/bin to the path. On my Linux box, the first command to set the path would seem to be in /etc/rc.d/rc.sysinit, which is one of the shell scripts invoked by the init process (inittab). Then, /etc/profile adds /usr/X11R6/bin. Finally, .bash_profile adds /home/ambler/bin.
  • PS1 - prompt string. Things that can be put in the prompt string include \h (hostname), \u (username), \w (absolute pathname of working directory), \W (name of working directory w/o path), \d (date), \t (time). See p.331 of A Practical Guide to Linux for more details. On my Red Hat boxes, the primary prompt string is set in the /etc/bashrc file. The prompt is also set in /etc/profile, but the setting in bashrc seems to take precedence. I also have a .bashrc file, which in turn runs /etc/bashrc, which sets the prompt. This means that the same prompt is used by xterm and rxvt in X sessions. On my Slackware box, the command line prompt is set in /etc/profile. The xterm and rxvt prompts are different, since I don't have a .bash_profile file which is run when an xterm is started. In fact, I don't know where the prompt is set in X sessions. It is not set by /usr/X11R6/lib/X11/app-defaults/XTerm.
  • PS2 - secondary prompt string.

Important Daemons and Startup Services

These are programs or processes which are run at boot time. Some remain in memory to execute various tasks when required (daemons). Most are started and stopped with scripts in the /etc/rc.d/init.d directory (see above). The exact contents of this directory will depend on which packages from a particular distribution are installed. For example, installing the Apache package will cause an httpd script to be placed in /etc/rc.d/init.d.

There are man pages on most of these. The Red Hat program tksysv (ntsysv is the non graphical version) allows root to automatically configure which of these are started automatically at boot time. The linuxconf program does the same thing, although I haven't tried it. The utility chkconfig is also designed to query and configure runtime services for different runlevels. The www.mandrakeuser.org site has a good page on common services/daemons, especially those included in recent versions of the Mandrake distribution.

A good source of information on daemons and services is the ``Linux Devices, Daemons, Services'' chapter of the CTDP (2000a) document.

  • amd - runs the automount daemon for remote filesystem mounting such as nfs
  • anacron - checks delayed `cron' tasks (see below) at boot time and executes them. Useful if you have cron jobs scheduled but don't run your machine all the time.
  • apmd - Advanced Power Management BIOS daemon. For use on machines, especially laptops, that support apm. Monitors battery status and can shut down the system if power is too low.
  • arpwatch - keeps watch for ethernet IP address pairings that are resolved using the ARP protocol.
  • atd - runs jobs queued by `at'
  • autofs - control the operation of automount daemons, used to mount and unmount devices on demand
  • bootparamd - allows computers to boot from a Linux machine using the BOOTP network protocol. A server process that provides information to diskless clients necessary for booting
  • crond - automatic task scheduler. Manages the execution of tasks that are executed at regular but infrequent intervals, such as rotating log files, cleaning up /tmp directories, etc.
  • cups - daemon for print services under the Common Unix Printer System, a replacement for lpd
  • dhcpd - implements the Dynamic Host Configuration Protocol (DHCP) and the Internet Bootstrap Protocol (BOOTP). Used to lease out IP addresses to remote machines.
  • drakfont - font server in Mandrake
  • fetchmail - daemon to fetch mail at regular intervals from mail servers
  • ftpd - ftp server daemon
  • gated - routing daemon that handles multiple routing protocols and replaces routed and egpup
  • gpm - useful mouse server for applications running on the Linux console.
  • httpd - the Apache webserver hypertext transfer protocol daemon
  • identd - The identd server provides a means to determine the identity of a user of a particular TCP connection. Given a TCP port number pair, it returns a character string which identifies the owner of that connection on the server's system.
  • inetd - listens for service requests on network connections, particularly dial-in services. This daemon can automatically load and unload other daemons (ftpd, telnetd, etc.), thereby economizing on system resources. In the latest version of Red Hat (7.0 at the time of writing), it has been replaced by xinetd. A partial list of services controlled by inetd is listed below. Under many distributions, inetd will execute scripts in the file /etc/inetd.conf.
  • innd - Usenet news server daemon
  • ipchains - daemon for packet forwarding. Used for configuring a gateway/firewall.
  • isdn provides ISDN network interfacing services
  • isdn4linux - for users of ISDN cards
  • kerneld - automatically loads and unloads kernel modules
  • keytable - loads the appropriate keyboard map from /etc/sysconfig/ keyboard
  • kheader -
  • kudzu - detects and configures new or changed hardware during boot
  • linuxconf - ``startup hook'' needed for the linuxconf system configuration tool
  • lpd - line printer and print spooler daemon
  • mcserv - server program for the Midnight Commander networking file system. It provides access to the host file system to clients running the Midnight file system (currently, only the Midnight Commander file manager). If the program is run as root the program will try to get a reserved port otherwise it will use 9876 as the port. If the system has a portmapper running, then the port will be registered with the portmapper and thus clients will automatically connect to the right port. If the system does not have a portmapper, then a port should be manually specified with the -p option (see below).
  • mysql - database server daemon
  • named - provides DNS services
  • netfs - network filesystem mounter. Used for mounting nfs, smb and ncp shares on boot.
  • network -activates all network interfaces at boot time by calling scripts in /etc/sysconfig/network-scripts
  • nfsd - used for exporting nfs shares when requested by remote systems
  • nfslock - starts and stops nfs file locking service
  • numlock - locks numlock key at init runlevel change
  • pcmcia - generic services for pcmcia cards in laptops
  • portmap - needed for Remote Procedure Calls
  • postfix - mail transport agent which is a replacement for sendmail. Now the default on desktop installations of Mandrake.
  • postgresql - database server daemon
  • random - random number generating daemon, related to security and encryption
  • routed - manages routing tables
  • rstatd - kernel statistics server. Allows users on a network to get performance statistics for any connected machine.
  • rusersd - provides services that allow users to find one another over the network
  • rwalld - allows users to use rwall to write messages on remote terminals
  • rwhod - server which maintains the database used by the rwho(1) and ruptime(1) programs. Its operation is predicated on the ability to broadcast messages on a network.
  • sendmail - mail transfer agent. This is the agent that comes with Red Hat. Others, such as smtpd, are not included.
  • smb - needed for running SAMBA
  • snmpd - provides Simple Network Management Protocol support
  • sound - daemon for managing sound
  • squid - web page proxy server daemon
  • syslogd - manages system log files
  • smtpd - Simple Mail Transfer Protocol, designed for the exchange of electronic mail messages. Several daemons that support SMTP are available, including sendmail, smtpd, rsmtpd, qmail, zmail, etc.
  • tcpd - from the tcp_wrappers package. Intercepts requests normally handled by inetd and filters them through the files hosts.allow and hosts.deny files, which can restrict access to services based on type of service, origin of request, destination, etc. Requests are intercepted because calls to particular services are replaced with calls to tcpd in /etc/inetd.conf.
  • telnetd - telnet server daemon
  • usb - daemon for devices on Universal Serial Bus
  • xfs - X font server
  • xinetd - more modern replacement for inetd. It apparently allows for similar kinds of access filters to the ones used by tcpd in conjunction with inetd. xinetd replaces inetd as the default network services daemon in Red Hat 7.0.
  • xntpd - Network Time Protocol daemon. Provides a means to syncronize time over the network.
  • webmin - daemon for webmin web-based system administration program
  • ypbind - NIS binder. Needed if computer is part of Network Information Service domain.

Alphabetical List of Principal Commands

In the following command list, the distinction between upper case and lower case letters is important. Most of the commands are utilities that are run by invoking their own executable files. In some cases, they are commands which are internal to a shell such as bash (shell builtins). The shell builtins are indicated. There are now many utilities that are included with either the Gnome or the KDE desktop environments, so many that it would be difficult to include them all here. In many cases, they duplicate the functionality of one or more of the programs listed below. I would suggest consulting the online documentation for these packages.

Another good source of information on commands is the CTDP (2000b) document. There are different chapters which group commands in different categories.

On rpm-based systems, to find out which package owns the command foo (where foo is a standalone executable), use the command rpm -q -f foo.

  • a2p - translation utility from awk to Perl
  • a2ps - translation utility from ``any'' to Postscript
  • ac - print statistics concerning user connect time
  • access - determine whether a file can be accessed
  • acroread - Adobe utility for viewing pdf files
  • adduser - used by root to add user to system
    • usage: adduser userid
  • afio - utility to copy or restore files to an archive file. This utility is not part of the basic internal and external programs under Red Hat Linux. It is available as an rpm on the rpmfind.net site.
  • agetty - enables login on terminals. See getty, mgetty, and uugetty.
  • ali - list mail aliases
  • alias - assign name to specified command list. This is actually a shell builtin. On my Red Hat system, I have global alias commands in my /etc/bashrc file and in my /.bashrc file. On my Slackware box, I have made the rm command a little bit safer with
    • alias rm='rm -i'

so that you can't recursively delete your /dev directory without telling the system you're sure you want to do it.

  • alien - utility to convert to and from different Linux package formats. Can handle Debian (deb), Stampede (stp), Red Hat (rpm) and Slackware (tgz) packages.
  • apropos - display command names based on keyword search
    • usage: apropos keyword
  • apsfilter - printer filter called by lpd to deal with printing different types of files. This is a fairly sophisticated print filter. It is not set up by default in Slackware. It used to be available on the second cd of a Slackware distribution. Since 4.0, it is available as one of the main packages in the ap set. Read the mail that the installation program sends to the root user. I believe that Red Hat uses its own printer filters.
  • ar - create, modify and extract from archives
  • arch - print machine architecture type
  • as - the portable GNU assembler
  • asapm - AfterStep laptop advanced power management utility
  • ash - a shell, a very simple shell program sometimes used on boot diskettes since it takes up much less space than bash, tcsh, zsh, etc.
  • asload - AfterStep cpu load monitor
  • asmail - AfterStep mail checking utility
  • asmodem - AfterStep utility to monitor modem status
  • aspell - a spell checking program along the lines of ispell
  • aspostit - X Window postit note utility
  • at - executes a shell script at specified time. Use atq to show pending jobs, and atrm to remove jobs from the queue.
    • usage: at time
    • or: at -f file time
  • atq - shows pending jobs queued by at. If run by root, shows everybody's pending jobs.
  • atrm - removes pending jobs queued by at. Use atq to determine the identities of various jobs.
    • usage: atrm job
  • awk - searches for and process patterns in a file
  • banner - print banner to standard output. Syntax is
    • banner [option] [characters]
  • bash - Bourne again shell. This is the default shell in the Red Hat installation.
  • batch - queue, examine, or delete jobs for later execution. See at.
  • bc - a language (compiler) similar to C, with unlimited precision arithmetic
  • bg PID - send process with pid ``PID'' to the background. This is the same as executing [Ctrl]z while interacting with the running process. This is a shell builtin.
  • bh - puts a job in the background. This is a shell builtin.
  • biff - mail notification utility. Notifies user of mail arrival and sender's name.
  • bind - displays or redefines key bindings. This is a shell builtin.
  • bison - parser generator similar to yacc
  • bru - a powerful backup utility program. Commercial. Demonstration versions are often included with Linux distributions such as Red Hat.
  • bsh - equivalent to ash
  • bunzip2 - used to uncompress files compressed with bzip2
  • byacc - parser generator
  • bzip2 - compresses with algorithm different from gzip
  • bzless - view bzipped files
  • c++ - invokes GNU C and C++ compiler
  • cal - displays a 12-month calendar for the given year or a one-month calendar of the given month
    • usage: cal month year
  • cat - combine, copy standard input to standard output. Used to join or display files.
  • cd - change working directory. This is a shell builtin in bash, tcsh and zsh.
  • cdplay - command line utility for playing audio cds
  • cfdisk - similar to fdisk, but menu-driven
  • chat - used to interact with a modem via a chat script
  • chgrp - changes group associated with file. Can be used to change the group associated with subdirectories and files of a directory.
    • usage: chgrp group files
    • or: chgrp -R group files
  • chkconfig - Query or update system services/daemons for different runlevels. Manipulates the various symbolic links in /etc/rc.d. This utility is included with many rpm-based distributions such as RedHat and Mandrake. It is designed to work with System V initialization scripts. Graphical tools for configuring system services include ntsysv, tksysv and ksysv (the latter is a KDE utility).
  • chmod - set permissions (modes) of files or directories. A value of 4 is used for read permission. A value of 2 is used for write permission. A value of 1 is used for execute permission. See umask for default file permissions upon file creation. Chmod can also be used to change the suid bit on files. The syntax for the symbolic version is
    • chmod [options] who operation permission file-list

The syntax for the absolute version is

    • chmod [options] mode file-list

To set the uid to the owner's permissions, use

    • chmod u +s file-name

To set the uid to the group's permissions, use

    • chmod g +s file-name

There are lots of security issues related to allowing a program to have root's permissions when run by an ordinary user. I don't pretend to understand all of these issues.

  • chown - changes ownership of a file. Can be used recursively.
    • usage: chown userid files
    • or: chown -R userid files
  • chsh - change default shell
  • ci - creates or records changes in an RCS file
  • clear - clear screen command
  • cmp - compares two files for differences
    • usage: cmp file1 file2
  • co - retrieves an unencoded version of an RCS file
  • comm - compares sorted files
  • configure - automatically configures software source code
  • color-xterm - color xterm program. Under Red Hat, this is just a link to xterm-color.
  • control-panel - graphical system configuration tool under Red Hat.
  • cp - copies on or more files. Recursive copying is one simple way of archiving part of a directory structure. Use the command as follows:
    • cp -r /sourcedirectory /targetdirectory
  • cpio - direct copy of files to an output device. Allows creation of archive file spanning multiple diskettes. Allows one directory structure to be mirrored elsewhere on the partition or on another partition. In order to back up an entire directory structure on diskettes, cd to the directory and use the following command:
    • find . -depth -print | cpio -ov > /dev/fd0

To restore from diskettes, use:

    • cpio -iv < /dev/fd0

The cpio command will prompt the user to insert more diskettes as they are needed. The command for mirroring a directory structure is the following:

    • find . -depth -print | cpio -pv /destinationdirectory

This copies the working directory and its contents, including subdirectories, into /destinationdirectory. In order to copy an individual file which is larger than a floppy, use:

    • find . -name nameoffile -print | cpio -iv > /dev/fd0
  • cpkgtool - Slackware tool to install, uninstall and query packages. Front end to installpkg, removepkg, makepkg. This is the graphical version that uses ncurses.
  • cpp - GNU C-compatible compiler preprocessor
  • crontab - schedules command to run at regularly specified time
  • csh - run C shell
  • csplit - separate files into sections. See also split.
  • cvs - manages concurrent acces to files in a hierarchy. Stands for concurrent version system. Is built on RCS. It stores successive revisions of files efficiently and ensures that access to files by multiple developers is done in a controlled manner. Useful when many developers are working on the same project.
  • cut - selects characters or TAB-separated fields from lines of input and writes them to standard output
  • date - displays or sets date and time
    • usage: date
    • or: date date
  • dd - direct copy of file from one device to another. Can be used to make copies of boot or root diskettes for installing Linux. It can be used, for example, to make and exact copy of a floppy disk, as follows. First, place the diskette to be copied in the floppy drive. Then,
    • dd if=/dev/fd0 ibs=512 > floppy.copy
    • Replace the diskette with a fresh diskette.
    • dd if=floppy.copy bs=512 of=/dev/fd0

The ibs and bs options specify the block sizes for input and for both input and output. A boot disk image can be directly copied to a floppy using the second of the two dd commands above.

  • declare - declares attributes for a variable (same as typeset). This is a shell builtin.
  • df - displays capacity and free capacity on different physical devices such as hard drive partitions and floppy drives that are mounted on the file system. Gives free space in blocks. With the (undocumented) option -h, the program gives free space in Mb or Gb. This is useful for those accustomed to thinking of the capacity of a high-density 3.5 inch diskette as 1440k.
  • diff - displays differences between two files
    • usage: diff file1 file2
  • diff3 - compares three files and reports on differences
  • dip - used to set up a SLIP or PPP connection. It can be used to set up an outgoing SLIP connection or an incoming connection.
  • diplogin - used for setting up incoming dip connections. See the man page for dip.
  • dir - a variation of the GNU ls command that defaults to printing file names in columns
  • dircolors - set colors for GNU ls command. In Slackware, this command is run by the /etc/profile script. Then, whenever xterm is run with the -ls (login shell) option, ls displays different colors for different types of files. Typical usage is eval `dircolors -b`. In Red Hat and Mandrake, I get color directories by aliasing the ls command (see below).
  • display - set display for output of programms under X Windows. Can be used to run a program on a remote machine while displaying the output on a local machine. The remote machine must have permission to send output to the local machine. This is actually an environment variable. See the more detailed discussion in connection with the xhost command below.
  • dmesg - displays messages from /var/log relative to the most recent boot
  • dos - invoke the DOSEMU DOS emulator
  • du - displays information on disk usage. The command
    • du / -bh | less

will display detailed disk usage for each subdirectory starting at root, giving files sizes in bytes.

  • dumpkeys - print information about the keyboard driver's translation tables to standard output
  • dvilj - send a dvi file to a Laserjet printer. There are specialized versions for individual models of Laserjet printer.
  • dvilj2p - specialized version of dvilj for the IIp series of printers. See above.
  • dvips - send a dvi file to a Postscript printer, to a Postscript capable Laserjet printer, or to a file (with the -o option). There is a switch to print only a subset of the pages, and another switch to print in landscape mode. Use -t landscape, which is one of the arguments to the paper type switch. If you have one page of a document that is a wide table, and you wish to print this in landscape mode, use
    • dvips filename -pp pagenumber -t landscape
  • e2fsck - check an ext2 filesystem. The syntax is
    • e2fsck /dev/devicename

where the filesystem is on /dev/devicename. The device should not be mounted, and this program must be run as root.

  • echo - write arguments to standard output. One use is to print out information about environment variables, as in
    • echo $PATH - list paths to search
    • echo $HOME or echo ~ - list name of home directory

This is a shell builtin.

  • editres - a dynamic resource editor for X Toolkit applications. Allows the user to change X resources for individual applications.
  • efax - fax program
  • efix - convert between fax, text, bit-map and gray-scale formats
  • egrep - search files for lines that match regular expressions. Runs faster than grep and fgrep.
  • elm - an interactive mail system
  • elvis - a version of the vi text editor
  • emacs - screen oriented text editor
  • env - desplay the current environment or set a variable equal to a new value
  • eval - scans and evaluates the command line. See dircolors command. This is a shell builtin.
  • ex - interactive command-based editor. The man page lists it as being the same as vim, an improved version of vi.
  • exec - system call which creates a subshell to execute a binary or a script. This is a shell builtin.
  • execve - a variation of the exec command.
  • exit - exit a shell. This is a shell builtin.
  • expand - convert tabs in files to spaces and write to standard output
  • expect - a program that ``talks'' to other interactive programs according to a script. Following the script, Expect knows what can be expected from a program and what the correct response should be. An interpreted language provides branching and high-level control structures to direct the dialogue. In addition, the user can take control and interact directly when desired, afterward returning control to the script.
  • export - place the value of a variable in the calling environment (makes it global). This is a shell builtin.
  • expr - utility evaluates an expression and displays the result
  • f2c - FORTRAN to C translator
  • f77 - FORTRAN 77 compiler
  • false - null command that returns an unsuccessful exit status
  • fax - simple user interface to efax and efix programs
  • fc - views, edits, and executes commands for the history list. This is a shell builtin.
  • fdformat - low level format of a floppy device
  • fetchmail - retrieve mail from a remote mail server and pass it to local SMTP agents on the local machine
  • fdisk - used to partition hard drives
    • usage: fdisk device
  • fg PID - bring a background or stopped process with pid ``PID'' to the foreground. This is a shell builtin. If only one process is running in background mode, fg with no argument is sufficient to bring it to the foreground
  • fgrep - search for patterns in files
  • file - displays classification of a file or files according to the type of data they contain
  • find - find files according to a large variety of search criteria. The find command that I use the most is
    • find . -name filename -print

in order to find files matching a particular name on the working directory and all subdirectories. Find can be incredibly powerful, but it is incredibly obscure.

  • finger - display information about a specified userid or userids
  • fmt - simple text formatting utility. Tries to make all nonblank lines nearly the same length.
  • fold - break lines of specified files so they are no wider than a specified lengths
  • fortune - available in the bsdgames package in Slackware and other distributions. Put a call to fortune in /etc/profile and get something inspirational or amusing every time you fire up an xterm as a login shell.
  • free - gives used and free memory on system along with other useful information
  • fromdos - takes a DOS text file from stdin and sends a UNIX file to stdout.
  • fsck - file system check and repair
  • ftp - file transfer over network
  • g++ - C++ compiler
  • g77 - GNU Fortran 77 compiler
  • gawk - GNU awk, mostly for processing delimited text files
  • gcc - invoke C, C++ compiler
  • getipts - parses arguments to a shell script. This is a shell builtin.
  • getkeycodes - print kernel's scancode-to-keycode mapping table
  • ghostscript - set of printing utilities. It seems to be obligatory to have this if a TEX installation such as teTEX is installed. How they communicate with one another is somewhat obscure.
  • ghostview - Aladdin ghostscript interpreter/previewer
  • gimp - image manipulation and paint program
  • glint - Red Hat graphical front end for the rpm package installer and manager.
  • grep - used to find a string within a file. The -i option returns matches without regard to case. The -n option means that each line of output is preceded by file name and line number. The -v option causes non-matched lines to be printed.
    • usage: grep pattern files
    • or: grep -i pattern files
    • or: grep -n pattern files
    • or: grep -v pattern files
  • groupadd - create a new group on the system
  • groups - shows which groups you are in
  • grub - Gnu grand unified bootloader. Can be used instead of lilo to boot multiple operating systems. I encountered a couple of snafus trying to install grub on my home machine after installing Mandrake 8.0 and choosing the lilo bootloader during the initial install. The Mandrake installation program set up /boot/vmlinuz as a symlink to the actual kernel, vmlinuz-2.4.3-20mdk. It took me a while to figure out that grub doesn't understand symbolic links. The documentation suggests installing grub on a diskette using the ``dd'' command. This refused to work, but
    • grub-install '(fd0)'

did work. The single quotes are necessary. The files necessary to run grub are normally located in /boot/grub. Once the file menu.lst has been edited and appropriated entries added to boot the different operating systems on one's hard disk(s), the following sequence of commands can be used to install grub in the master boot record (MBR) sector of the hard disk:

    • root (hd0,x)
    • setup (hd0)

Here, the x should be replaced by the partition where the /boot/grub directory is located, which is probably the root partition of the Linux system. Note that grub has its own conventions for naming devices and numbering partitions, so that for example a partition which is called hda6 under Linux will be called (hd0,5) by grub.

  • grub-install - command to install grub on the hard drive (or floppy drive).
  • gunzip - used to uncompress files compressed with gzip
  • gv - PostScript and PDF previewer, based on ghostview
  • gvim - see vi
  • gzexe - compresses executables
  • gzip - used to compress or decompress files
  • halt - shut down system as root, without reboot, immediately
  • hash - remembers the location of commands in the search path. This is a shell builtin.
  • head - displays first part of a file
  • history - command for viewing and manipulating the shell command history list
  • host - look up host names using domain server
  • hostname - used to get or set hostname. Typically, the host name is stored in the file /etc/HOSTNAME.
  • hwclock - used to query and set the hardware clock
  • hylafax - commercial fax program
  • id - display userid and groupid
  • inetd - daemon which starts up other daemons on demand. Configured in /etc/inetd.conf.
  • ifconfig - display (as root) information on network interfaces that are currently active. First ethernet interface should be listed as eth0, second as eth1, etc. First modem ppp connection should be listed as ppp0, etc. The ``lo'' connection is ``loopback'' only.
  • ifdown - shut down the network interface
  • ifup [interface_name] - start up the interface
  • info - display system information. This is the GNU hypertext reader.
  • init - the mother of all processes, run at bootup, executes commands in /etc/inittab. Can be used (with root privileges) to change the system run level.
    • usage: init run_level
  • insmod - used (by root) to install modular device drivers
  • installpkg - Slackware command to install one of the packages from the program sets
  • intr - interrupt key, usually [Ctrl-C]
  • ispell - checks files for spelling errors
    • usage: ispell files
  • jed - programmer's file editor. Behaves like emacs. Has modes for TEX, FORTRAN, C, etc.
  • jobs - displays list of current jobs in the background. This is a shell builtin.
  • joe - simple WordStar-like text editor. It can be invoked in emacs emulation mode with jemacs and in WordStar emulation mode with jstar.
  • jove - Joseph's Own Version of Emacs. A simple emacs clone.
  • kbd_mode - print current keyboard mode
  • kernelcfg - GUI to add/remove kernel modules (as root in X terminal).
  • kerneld - kernel daemon, a process that stays in memory and does all sorts of useful stuff, like automatic loading of device driver modules
  • kikbd - a utility program that comes with KDE that allows users to switch on the fly among different international keyboards. It can be used under different window managers than kfm.
  • kill - sends a signal to (especially to terminate) a job or process. This is a shell builtin in bash, tcsh and zsh.
  • killall - kill processes by name. Kill all processes which are instances of the speciffied program. Also used to send signals to processes or restart them.
  • killall5 - kill all processes except the ones on which it depends
  • last - generate a listing of user logins
  • lastlog - prints the last login times of all users
  • latex - compile a LATEX file
  • ldconfig - creates the necessary links and cache (for use by the run-time linker, ld.so) to the most recent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories (/usr/lib and /lib). Ldconfig checks the header and file names of the libraries it encounters when determining which versions should have their links updated. Ldconfig ignores symbolic links when scanning for libraries.
  • ldd - list the shared libraries on which a given executable depends, and where they are located
  • leave - display reminder at specified time
  • less - Linux alternative to ``more'' command. Displays text files, one screenful at a time. When less pauses, there is a large number of available commands to tell it what to do next. One can scroll both forwards and backwards.
  • let - evaluates a numeric expression. This is a shell builtin.
  • lilo - installs boot loader on the boot sector of a hard drive, of a diskette, or in another location. My 486 has a hard drive that is too large for the machine's BIOS, so I have to boot from a floppy. To create a boot diskette, I do the following (as root):
    • /sbin/fdformat /dev/fd0H1440
    • /sbin/mkfs.ext2 /dev/fd0
    • mount -t ext2 /dev/fd0 /mnt/floppy
    • cp -dp /boot/* /mnt/floppy
    • /sbin/lilo -C /etc/lilo.flop

The -C option to lilo has lilo use the lilo.flop file instead of the default lilo.conf.

  • linuxconf - interactive tool for configuring Linux system. Uses X if loaded. This is a Gnome tool. It comes with my Red Hat distribution, and is not included with Slackware. It would seem to be the easiest way to configure Linux under Slackware. Version 1.15 is available for Slackware. There is a pretty good introduction to the use of linuxconf in the Red Hat 5.2 installation manual, which is available online at their web site.
  • ln - creates a link to a file. Used to create hard links and, with the -s option, symbolic links which can link files on different disk partitions. The syntax is
    • ln [options] source [dest]
  • locate filename - find the file name which contains the string ``filename''. The syntax is easier than the find command.
  • lock - temporarily lock terminal
  • lockfile - create semaphore file(s), used to limit access to a file
  • - log in to system
  • logname - consult /etc/utmp for user's login name
  • logout - execute logout as individual user and bring up login: prompt
  • look - look for strings in files
  • lpq - show print jobs that are waiting
  • lpr - send file to be printed
  • lprm - cancel a job from print queue
  • ls - list directory contents. To get colored directory listings under Red Hat, Mandrake, etc., use
    • ls -color

To get this all the time, add

    • alias ls='ls -color=auto'

to .bashrc. The following command

    • alias ls='ls -Fskb -color=auto'

will give directory listings in color, with file sizes in kilobytes, and append a character to the file to indicate its type.

  • lsattr - list attributes of files in ext2 file system
  • lsmod - used (by root) to show kernel modules currently loaded
  • lspci - utility to display information on pci buses and hardware devices attached to them. Part of the pciutils package that comes with many Linux distributions.
  • lspnp - utility to display information about pnp devices. Part of the pcmcia or kernel-pcmcia package, depending on the distribution.
  • m4 - an implementation of the traditional UNIX macro processor. It can be used with the sendmail configuration package in Red Hat (and Slackware) to generate a sendmail.conf configuration file without having to edit the configuration file directly.
  • magicfilter - general purpose printer filter. See apsfilter above. apsfilter is the printer filter that comes with the Red Hat and Slackware distributions.
  • mail - sends or reads electronic mail
  • make - keeps a set of programs current. This is a utility that helps when developing a set of programs. It works by executing a script called makefile, Makefile or GNUmakefile in the working directory. It is very often used in combination with configure when compiling and installing noncompiled software packages.
  • makebootdisk - command in Slackware to do just what the name says
  • MAKEDEV - executable script to make device files on /dev
  • makeswap - configures swap space
  • man - displays information from online Unix reference manual
  • manpath - attempt to determine path to manual pages
  • mc - Midnight Commander file manager and visual shell
  • mesg - enables/disables reception of messages
  • minicom - terminal program
  • mkdir - create a directory
  • mkfs - create a file system (format) on a device or partition. Should be invoked after lowlevel formatting of the disk using fdformat. It has several versions which are all links to the basic program, such as mkfs.ext2 and mkfs.msdos.
  • mkswap - creates a Linux swap space on the specified hard disk parition (root privileges neede)
    • usage: mkswap device
  • more - list file contents, stopping after each full screen
  • mount -t [fstype] [device] [mountpoint] - mount device using filesystem of type [fstype] with device name [device] at the location [mountpoint] in the filesystem directory tree
  • mount -a - mount all filesystems according to the specifications in /etc/fstab
  • mouseconfig - mouse configuration utility under Red Hat. Located in /usr/sbin.
  • mpage - print multiple pages per sheet on a Postscript printer. Can also be used to print a page in landscape mode.
  • Mtools - package of MS-DOS utilities. Includes the following commands.
    • mcd - changes working directory on DOS disk
    • mcopy - copies DOS files from one directory to another
    • mdel - deletes DOS files
    • mdir - lists contents of DOS directories
    • mformat - adds DOS formatting information to a disk
    • mtype - displays contents of a DOS file

The default device for execution of these commands is /dev/fd0 and can be referred to as ``a:''.

  • mv - moves (renames) files
  • netconf - used (as root) to set up network
  • newaliases - rebuilds the /etc/aliases database used by sendmail. Must be rerun every time /etc/aliases is modified for the changes to take effect.
  • newgrp - similar to login. Changes user's identification
  • nice program_name - sets the priority of the program ``program_name''.
  • nm - lists the symbols from object files objfile. If no object files are given as arguments, nm assumes `a.out'.
  • nohup - runs a command that keeps running after logout. The command is in principle immune to hangups, and must have output to a non tty. According to Linux in a Nutshell, this is necessary only in the Bourne shell, since modern shells preserve background processes by default.
  • ntsysv - run level editor under Red Hat. This is the equivalent of tksysv, but does not require a graphical interface.
  • nxterm - color xterm program. The man page for nxterm under Red Hat brings up the same page as xterm.
  • od - dumps contents of a file
  • passwd - change login password
  • paste - joins corresponding lines from files
  • patch - updates source code. Attempts to update a file from a file of change information, or pathces, created by diff.
  • pathchk - determine validity and portability of filenames
  • pdflatex - part of the pdftex program suite. Produces pdf output from a LATEX file.
  • pdftex - produces pdf output from a TeX file. See also pdflatex. This program is part of the tetex 0.9 distribution that is included with Red Hat 5.2 and above, and with Slackware 4.0 and above. It is also available as a separate program.
  • perl - practical extraction and report language
  • pg - display data one screenful at a time
  • pico - simple screen oriented text editor. It is included as part of the Pine program.
  • ping - check if Internet computer is responding. Can also measure the time it takes the queried computer to respond.
  • pkgtool - Slackware tool to install, uninstall and query packages. Front end to installpkg, removepkg, makepkg. The cpkgtool is the ncurses graphical version of this program.
  • popclient - retrieve mail via the Post Office Protocol. Supports POP2 and POP3.
  • popd - pops the top directory of the directory stack and uses cd to change to that directory. This is a shell builtin.
  • pr - paginates files for printing
  • printenv - display list of environment variables
  • printtool - run (as root) in an X terminal to configure your printer(s)
  • ps - displays status of processes. Use the -a option for processes for all users. Use the -x option to include processes not attached to a terminal.
  • pstree - display processes in the form of a tree structure. Killing a parent process will also kill all the children and their descendants.
  • pushd - pushes the argument onto the top of the directory stack and uses cd to change to that directory. This is a shell builtin.
  • pwd - print absolute path of working directory. This is a shell builtin.
  • pwchk - checks the integrity of password and shadow files
  • pwconv - converts passwords to the shadow password format
  • pwunconv - unconverts passwords from the shadow password format. Generates a standard Unix password file.
  • python - interpreted, interactive, object-oriented programming language
  • rcp - copy one or more files to or from remote computer. The syntax is poorly explained in the documentation that I have, including the man pages. Usage is:
    • rcp filename username@remotehost:path

The user's home directory on the remote system must contain the file .rhosts with a list of users (preceded by the full domain name or exact IP address of their machine) with access privileges.

    • localhostname username
  • rcs - creates or changes the attributes of an RCS file. Stands for Revision Control System.
  • rdev - query/set image root device, swap device, RAM disk size, or video mode in kernel
  • read - reads line from standard input. This is a shell builtin.
  • readonly - declares a variable to be read only. This is a shell builtin.
  • reboot - in Slackware, reboots the system. Seems to be equivalent to shutdown -r now in generic Linux.
  • renice program_name - resets the priority of process ``program_name''.
  • reset - used to reset the screen characteristics. This is useful if the screen gets messed up from, for example, trying to display a binary file in an xterm.
  • return - exits from a function. This is a shell builtin.
  • rlog - prints a summary of the history of an RCS file
  • rlogin - log in to remote computer. The general syntax is as follows, using the UQAM Nobel machine as an example:
    • rlogin -l userid nobel.si.uqam.ca

The remote computer must recognize the local user and the local machine. See the rcp command for how to set up the .rhosts file on the remote machine.

  • rm - remove files or directories. With the -r (recursive) option (very dangerous!), can be used to remove the contents of a specified directory including all subdirectories.
  • rmail - interpret and handle remote mail received via uucp
  • rmdir - remove empty directories
  • rmmod - used to remove modular device drivers
  • route -n - show routing table. The n option returns numerical addresses rather than names.
  • rpm - invokes the Red Hat package manager in command line mode. I often use this command in query mode to query packages about what files they contain and to find out which package owns a particular file. Examples are
    • rpm -qil foo. Gives package information and a file list for the package foo.
    • rpm -qfil foo. Gives package information and a file list for the package that owns the file foo. Foo must be in the working directory, or the full path to foo must be specified.

We need to find out about installing the rpm package on a Slackware box. It's probably better to use a package converter such as alien.

  • rpm2tgz - an extremely useful utility on Slackware systems that converts rpm packages to tgz format. They can then be installed using the installpkg command (or pkgtool).
  • rsh - execute shell command on a remote computer. See rcp and rlogin.
  • rstat - summarize host's status: uptime, load averages, and current time
  • ruptime - show host status of local machines
  • rusers - list who is logged on local machines
  • rwall - write to all users over a network
  • rwho - show who is logged in on a LAN. The rwho service must be enabled for this command to run. If it isn't, run ``setup'' as root. I don't understand this last remark, which comes from ``Linux Newbie Administrator Guide''.
  • rxvt - a terminal program similar to xterm, but which has less features and uses less memory
  • sed - edits a file (not interactively). Also a tool for processing text files.
  • set - set or display value of shell variables. This is a shell builtin. The command
    • set | less

prints the current user environment, giving the values of currently defined variables.

  • setenv - set or display value of environment variables
  • setserial - used by root to configure a serial port
  • setterm - set terminal attributes for a virtual console
  • setuid - set the id of a program when it is run. Used, for example, to give root privileges to a program run by an ordinary user. This is actually done by running the chmod program as root. See the chmod command for the syntax.
  • setup - Slackware program to set up program sets and configure system. Setup devices and file systems, mount root file system
  • sh - standard UNIX shell. On Linux, just another name for bash.
  • shift - promotes each command-line argument. This is a shell builtin.
  • showmount - show information about an nfs server
  • shutdown - reboot or shut down system as root, after specified amount of time. With the -r option, reboot. With the -h option, halt the system.
    • usage: shutdown -r minutes
  • sleep - creates process that sleeps for specified interval
  • sliplogin - attaches a SLIP interface to standard input. Used to allow dialin SLIP connections.
  • sort - sorts and/or merge files
  • split - split file into specified number of segments
  • ssh - secure shell. Apparently has many of the same functionalities as rlogin, telnet, ftp, rsh, etc., with better security and encryption features. We may want to learn how to set this up and use it.
  • startx - front end to xinit in Linux. This is a script which starts up X clients and shuts down the X server on exit from the window manager.
  • startx -- :1 - start the next X window session on the display 1 (the default is opened on display 0). One can switch between different graphical displays using [Ctrl][Alt][F7], [Ctrl][Alt][F8], etc.
  • stty - sets or displays operating options for terminal
  • su - log in as another user, including root
  • sudo - allows individual users to have root permission to perform specified tasks
  • swapoff - disables swap disk
  • swapon - enables swap disk
  • symlinks - provide list of and information about symbolic links
  • sync - writes memory buffers to physical devices
  • systat - query host for system information
  • tac - print file in reverse
  • tail - displays the last part of a file
  • talk - visual communication program that copies lines from one terminal to that of another user
  • tar - file compression and archiving utility. I find the syntax of this command to be frustratingly opaque. The following works for me. To use this command to unzip gzipped tarballs in verbose mode, use
    • tar -xvzf filename.tgz

To create a tarball from files in a given directory and its subdirectories, use

    • tar -cvzf filename.tgz sourcename

Sourcename can be the name of a single file, a wildcard such as *, or the name of a subdirectory. There seem to be two different conventions concerning gzipped tarballs. One often encounters .tar.gz. The other popular choice is .tgz. Slackware packages use the latter convention. The command can also be used to archive a file, a group of files, or a directory (with its subdirectories) on tape or onto floppies. If the material to be archived exceeds the capacity of the backup medium, the program will prompt the user to insert a new tape or diskette. Use the following command to back up to floppies:

    • tar -cvf /dev/fd0 filename(s) or directoryname(s)

The backup can be restored with

    • tar -xvf /dev/fd0

Tar can be used for other things. To mirror all the files and subdirectories in from-stuff to to-stuff, use the commands

    • cd from-stuff
    • tar cf - . | (cd ../to-stuff; tar xvf -)

No tar file is ever written to disk. The data is sent by pipe from one tar process to another. This example is taken from Running Linux, p.177. To list the table of contents of a tar archive, use

    • tar tvf tarfile

To extract individual files from a tar archive, use

    • tar xvf tarfile files

where files is the list of files to extract. When extracting files, tar creates missing subdirectories underneath the current directory in which the cammand is invoked.

  • tcl - scripting language
  • tcsh - extended version of the C shell
  • tee - copy standard input to standard output and one or more files
  • telinit - used to change run level. Exact run level that corresponds to single-user, multi-user, and X levels depends on distribution.
  • telnet - remote login over network
  • test - evaluates an expression or compares arguments. This is a shell builtin in bash, tcsh and zsh.
  • tftp - user interface to TFTP protocol
  • time - displays times for the current shell and its children. This is a shell builtin. Strange, because there is also a /usr/bin/time program on my Red Hat system.
  • tin - Netnews reader
  • tkdesk - graphical desktop file manager for X
  • tksysv - graphical runlevel editor under Red Hat. Allows root to configure the services that are started at each run level.
  • tload - display system load average in graph format
  • top - dynamically displays process status
  • touch - update access and modification times of a file. If the file does not exist on disk, an empty file is created.
  • tr - translation utility that can be used, for example, to replace specified characters in a text file
  • trap - traps a signal. This is a shell builtin.
  • true - null command that returns a successful exit status
  • tset - initializes terminal
  • tty - shows special file that represents your terminal. Displays the terminal pathname.
  • type - displays how each argument would be interpreted as a command. This is a shell builtin.
  • typeset - declares attributes for a variaable (same as declare). This is a shell builtin.
  • ul - translate underscores to underlining
  • umask - establishes the file-creation permissions mask. Usage is
    • umask xyz

The system subtracts x, y and z from the owner, group and other file permissions that it would otherwise assign to new files. This is a shell builtin.

  • umount [device] - finish writing to the device and remove it from the active filesystem. The command umount -a will (re)mount all file systems listed in /etc/fstab.
  • unalias - remove name previously defined by alias. This is a shell builtin.
  • uname - displays information about the system. With no arguments, it displays the name of the operating system. With the -a option, it displays information about the operating system, the host name, and hardware.
  • uniq - displays lines of a file that are unique
  • unset - removes a variable or function. This is a shell builtin.
  • unzip - uncompress files compressed with the zip utility, compatible with DOS PKzip
  • updatedb - update file database used by locate command
  • uptime - shows the time, how long the system has been up, the number of users, and average load.
  • useradd - same as adduser
  • userdel - remove an account (as root). The user's home directory and undelivered mail must be dealt with separately.
  • users - prints list of users on the system
  • vdir - variant of the GNU version of the ls command. Defaults to printing out the long version of directory entries.
  • vi - standard screen oriented Unix editor
  • view - vi in read-only mode
  • vim - improved vi editor
  • vrfy - query remote host to verify the accuracy of an email address
  • w - display info about userids and active processes
  • wait - waits for a background process to terminate. This is a shell builtin.
  • wc - displays number of lines, characters and words in a file
  • Wharf - the AfterStep application dock module
  • whatis - display one-line summary of specified command
  • whereis - use to find utilities in standard locations
  • which - used to find utilities in search path. Will return the absolute directory path of the named utility program.
  • who - display information about currently logged in userids
  • whoami - display information about userid that is currently logged in
  • wish - front end to tk, an X window extension of tcl
  • workbone - console based cd player
  • workman - graphical cd player program
  • write - send messages to another local user
  • X - starts up the X server. Can be invoked with
    • X -quiet -query remotemachineaddress

in order to get a graphical login screen on the remote machine. See the discussion in connection with xdm below.

  • xadm - display advanced power management BIOS information
  • xargs - converts standard output of one command into arguments for another. This is one of those powerful but obscure commands. Xargs reads arguments from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial-arguments followed by arguments read from standard input. Blank lines on the standard input are ignored.
  • xbiff - graphical mail delivery notification utility
  • xcalc - simple calculator program
  • xclipboard - name says it all
  • Xconfigurator - Red Hat utility for configuring settings for X
  • xdm - used to start an X login session. This can be used to start a login session on a remote system. See the discussion on the following site:

See the man pages for X, xdm, and Xserver. As usual, the man pages are pretty obscure. The best single source seems to be the Xserver man pages. After X is configured, X needs to be started at bootup with the command (in /etc/rc.d/init.d/xterm):

    • X -quiet -query remotemachineaddress

If the address of a nameserver is not configured, then the numeric address of the remote machine rather than its name should be entered. If the machines are connected through ethernet cards and the net, then obviously basic networking has to be set up. Gnome and KDE come with their own versions of X display/login managers, called respectively gdm and kdm.

  • xdvi - view a dvi file compiled under LATEX
  • xedit - a simple text editor for X
  • xf86config - graphical configuration tool for X
  • XF86Setup - graphical configuration tool for X
  • xfd - display an available font in X. Creates a grid in an x-term with one character per rectangle.
  • xfig - utility for interactive generation of figures
  • xfm - graphical file manager for X
  • xhost - tell X server that remote computer has access to your machine and that you will use the remote computer. This can be used to set up remote X sessions. To set up a remote X session on the UQAM Nobel machine, run the following command on the local machine (one doesn't have to be root to do this)
    • xhost +nobel.si.uqam.ca

Then, log onto the remote machine using rlogin (see above) or telnet. Once logged in, use the following command to get the remote X server to open an X terminal on the local machine:

    • setenv DISPLAY localhostname:0 ; xterm &

This is valid for csh, which is the default login shell on Nobel. For ksh, (and I think bash) replace with

    • DISPLAY=localhostname:0
    • export DISPLAY ; xterm

Other X-based programs such as Netscape or Gauss (graphical version) can also be run on a remote machine with display on the local machine with little trouble. The local X server is the program that has all of the information concerning the properties of the graphics card and terminal, so it must be necessary to have X running on the local machine. The following should also work. After using xhost to give permission to the remote machine to display on the local machine, use

    • netscape -display localhostname:0.0

Question: can one start the X session on the local machine and then run a remote copy of a window manager?

  • xinit - start X Window. The command startx is a front end to xinit in Linux, including Slackware.
  • xload - displays a graphic of the system load
  • xlpq - graphical interface to print manager. This is included on one of the XFCE menus, but does not seem to be a part of the base Red Hat distribution.
  • xlsfonts - list fonts available under the X Window system.
  • xman - browsable command reference. Displays manual pages under X.
  • xmh - graphical front end under X to the nmh mail handling system. This program is part of the XFree86 package in Red Hat.
  • xmodmap - utility for modifying keymaps and pointer button mappings in X. Can be used to install a French Canadian keyboard. Download the Xmodmap.cf file from www.linux-quebec.org, and insert the command
    • xmodmap /etc/X11/Xmodmap.cf &

into the .xession (with xdm) or the .xinitrc (with startx) file.

  • xosview - displays bar graphs of system load, load average, memory usage, and swap usage
  • xpaint - simple paint program for X
  • xpdf - GPL'd utility for previewing dvi files. Doesn't seem to work too well on texts with a lot of math.
  • xplaycd - X Window audio cd player utility
  • xsetroot - utility to configure root window of an X terminal
  • xsysinfo - graphical display of load and memory usage
  • xterm - start an X Window terminal session
  • xterm-color - color version of xterm
  • xv - utility for viewing and manipulating many types of image files. This is a shareware program.
  • xvidtune - utility for fine tuning of monitor settings under X
  • yacc - parser generator
  • ytalk - multi-user program similar to talk
  • zcat - read one or more files that have been compressed with gzip or compress and write to standard output
  • zcmp - read compressed files and pass them to cmp
  • zdiff - read compressed files and pass them to diff
  • zgrep - read compressed files and pass them to grep
  • Zharf - AfterStep button panel module
  • zip - zip utility compatible with DOS PKzip
  • zless - view zipped files
  • zmore - print contents of compressed files one screen at a time
  • znew - uncompress Z files and recompress in .gz format

Eventually, we want to be able to distinguish between commands that are an intrinsic part of the kernel, commands that are executable binaries that come with every distribution of Linux, executable binaries that are not provided with all distributiions of Linux, and executable shell scripts. We also want to point out the typical location of these commands on different Linux distributions. Finally, we want to distinguish between shell commands and Linux commands.