The Linux Philosophy for SysAdmins





Last Updated on 09/20/2019 by dboth

Background

The Unix Philosophy is an important part of what makes Unix unique and powerful. Much has been written about the Unix Philosophy and the Linux philosophy is essentially the same as the Unix philosophy because of its direct line of descent from Unix.

The original Unix Philosophy was intended primarily for the system developers. Eric S. Raymond, in his seminal work, The Art of Unix Programming, set forth the tenets of the original philosophy as a set of guidelines for programmers.

The original Unix and Linux Philosophy was intended for the developers of those operating systems. Although System Administrators could apply many of the tenets to their daily work, many important tenets that address things unique to SysAdmins were missing. Over the years I have been working with Linux and Unix, I have formulated my own philosophy – one which applies more directly to the everyday life and tasks of the System Administrator. My Philosophy is based in part upon the original Unix and Linux Philosophy, as well as the philosophies of my mentors.

Having worked with Unix and Linux for over 20 years as of this writing, I have found that the Linux Philosophy has contributed greatly to my own efficiency and effectiveness as a SysAdmin. I have always tried to follow the Linux philosophy because my experience has been that a rigorous adherence to it, regardless of the pressure applied by a legion of Pointy-Haired-Bosses (PHB), will always pay dividends in the long run.

My book, The Linux Philosophy for SysAdmins, is the result of my SysAdmin approach to the Linux Philosophy. Much of this chapter is taken directly from that book. Because the name, “Linux Philosophy for SysAdmins,” is a bit long, most of the time I will refer to it in this book as the “Philosophy” for simplicity.

The structure of the Philosophy

There are three layers to the Linux Philosophy for System Administrators in a way that is similar to Maslow’s hierarchy of needs. These layers are also symbolic of our growth through progressively higher levels of enlightenment.

The bottom layer is the foundation – the basic commands and knowledge that we as SysAdmins need to know in order to perform the lowest level of our jobs. The middle layer consists of those practical tenets that build on the foundation and inform the daily tasks of the SysAdmin. The top layer contains the tenets that fulfill our higher needs as SysAdmins and which encourage and enable us to share our knowledge.

In the first and most basic layer of the Philosophy is the foundation. It is about “The Linux Truth,” data streams, Standard Input/Output (STDIO), transforming data streams, small command line programs, and the meaning of “everything is a file,” for example.

The middle layer contains the functional aspects of the Philosophy. Embracing the command line, we expand our command line programs to create tested and maintainable shell programs that we save and can use repeatedly, and even share. We become the “lazy admin” and begin to automate everything. We use the Linux filesystem hierarchy appropriately and store data in open formats. These are the functional portions of the Philosophy.

Figure 1: The hierarchy of the Linux Philosophy for SysAdmins.

The top layer of the Philosophy is about enlightenment. We begin to progress beyond merely performing our SysAdmin tasks and just getting the job done, our understanding of the elegance and simplicity in the design of Linux is perfected. We begin striving for doing our own work elegantly, keeping solutions simple, simplifying existing but complex solutions, and creating usable and complete documentation. We begin to explore and experiment simply for the sake of gaining new knowledge. At this stage of enlightenment, we begin to pass our knowledge and methods to those new to the profession and we actively support our favorite open source projects.

In my opinion it is impossible to learn about many Linux commands and utilities without learning about the structure and philosophy of Linux. Working on the command line requires such knowledge. At the same time, working on the command line engenders the very knowledge required to use it. If you use the command line long enough you will find that you have learned at least some about the intrinsic beauty of Linux without even trying. If you then follow your own curiosity about what you have already learned, the rest will be revealed.

Does that sound a bit Zen? It should because it is.

The Tenets

Here we look briefly at each of the tenets of the Linux Philosophy for the SysAdmin. This list must necessarily be terse and it cannot cover all aspects of each tenet. If you are interested in learning more, you should refer to my book, The Linux Philosophy for SysAdmins, for more information and the details of each tenet.

Data streams are a universal interface

Everything in Linux revolves around streams of data – particularly text streams. In the Unix and Linux worlds a stream is a flow text data that originates at some source; the stream may flow to one or more programs that transform it in some way, and then it may be stored in a file or displayed in a terminal session. As a SysAdmin your job is intimately associated with manipulating the creation and flow of these data streams.

The use of Standard Input/Output (STDIO) for program input and output is a key foundation of the Linux way of doing things and manipulating data streams. STDIO was first developed for Unix and has found its way into most other operating systems since then, including DOS, Windows, and Linux.

“This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.”

Doug McIlroy, Basics of the Unix Philosophy

STDIO was developed by Ken Thompson as a part of the infrastructure required to implement pipes on early versions of Unix. Programs that implement STDIO use standardized file handles for input and output rather than files that are stored on a disk or other recording media. STDIO is best described as a buffered data stream and its primary function is to stream data from the output of one program, file, or device to the input of another program, file, or device.

Data streams are the raw materials upon which the Core Utilities and many other CLI tools perform their work. As its name implies, a data stream is a stream of data being passed from one file, device, or program to another using STDIO.

Transforming Data Streams

This tenet explores the use of pipes to connect streams of data from one utility program to another using STDIO. The function of these programs is to transform the data in some manner. You will also learn about the use of redirection to redirect the data to a file.

Data streams can be manipulated by inserting transformers into the stream using pipes. Each transformer program is used by the SysAdmin to perform some transformational operation on the data in the stream, thus changing its contents in some manner. Redirection can then be used at the end of the pipeline to direct the data stream to a file. As has already been mentioned, that file could be an actual data file on the hard drive, or a device file such as a drive partition, a printer, a terminal, a pseudo-terminal, or any other device connected to a computer.

I use the term “transform” in conjunction with these programs because the primary task of each is to transform the incoming data from STDIO in a specific way as intended by the SysAdmin and to send the transformed data to STDOUT for possible use by another transformer program or redirection to a file.

The standard term for these programs, “filters,” implies something with which I don’t agree. By definition, a filter is a device or a tool that removes something, such as an air filter removes airborne contaminants so that the internal combustion engine of your automobile does not grind itself to death on those particulates. In my high school and college chemistry classes, filter paper was used to remove particulates from a liquid. The air filter in my home HVAC system removes particulates that I don’t want to breathe. So, although they do sometimes filter out unwanted data from a stream, I much prefer the term “transformers” because these utilities do so much more. They can add data to a stream, modify the data in some amazing ways, sort it, rearrange the data in each line, perform operations based on the contents of the data stream, and so much more. Feel free to use whichever term you prefer, but I prefer transformers.

The ability to manipulate these data streams using these small yet powerful transformer programs is central to the power of the Linux command line interface. Many of the Linux core utilities are transformer programs and use STDIO.

Everything is a file

This is one of the most important concepts that makes Linux especially flexible and powerful: Everything is a file. That is, everything can be the source of a data stream, the target of a data stream, or in many cases both. In this book you will explore what “everything is a file” really means and learn to use that to your great advantage as a SysAdmin.

“The whole point with “everything is a file” is … the fact that you can use common tools to operate on different things.”

Linus Torvalds in an email.

The idea that everything is a file has some interesting and amazing implications. This concept makes it possible to copy a boot record, a disk partition, or an entire hard drive including the boot record, because the entire hard drive is a file, just as are the individual partitions. Other possibilities include using the cp (copy) command to print a PDF file to a compatible printer, using the echo command to send messages from one terminal session to another, and using the dd command to copy ISO image files to a USB thumb drive.

“Everything is a file” is possible because all devices are implemented by Linux as these things called device special files which are located in the /dev/ directory. Device files are not device drivers, rather they are gateways to devices that are exposed to the user. I explore device special files in detail throughout my Linux self-study course, “Using and Administering Linux – Zero to SysAdmin

Use the Linux FHS

The Linux Filesystem Hierarchical Standard (FHS) defines the structure of the Linux directory tree. It names a set of standard directories and designates their purposes. This standard has been put in place to ensure that all distributions of Linux are consistent in their directory usage. Such consistency makes writing and maintaining shell and compiled programs easier for SysAdmins because the programs, their configuration files, and their data, if any, should be located in the standard directories. This tenet is about storing programs and data in the standard and recommended locations in the directory tree and the advantages of doing so.

As SysAdmins our tasks include everything from fixing problems to writing CLI programs to perform many of our tasks for us and for others. Knowing where data of various types are intended to be stored on a Linux system can be very helpful in resolving problems as well as preventing them.

The latest Filesystem Hierarchical Standard (3.0)8 is defined in a document maintained by the Linux Foundation. The document is available in multiple formats from their web site as are historical versions of the FHS. The Linux Foundation maintains documents defining many Linux standards. It also sponsors the work of Linus Torvalds.

Embrace the CLI

The Force is with Linux and the Force is the Command Line Interface – the CLI. The vast power of the Linux CLI lies in its complete lack of restrictions. Linux provides many options for accessing the command line such as virtual consoles, many different terminal emulators, shells, and other related software that can enhance your flexibility and productivity.

The command line is a tool that provides a text mode interface between the user and the operating system. The command line allows the user to type commands into the computer for processing and to see the results.
The Linux command line interface is implemented with shells such as Bash (Bourne again shell), csh (C shell), and ksh (Korn shell) to name just three of the many that are available. The function of any shell is to pass commands typed by the user to the operating system which executes the commands and returns the results to the shell.

Access to the command line is through a terminal interface of some type. There are three primary types of terminal interface that are common in modern Linux computers but the terminology can be confusing. These three interfaces are virtual consoles, terminal emulators that run on a graphical desktop, and an SSH remote connection. We will explore the terminology, virtual consoles, and one terminal emulator in Chapter 7. Several different terminal emulators are covered in Chapter 14.

Be the Lazy SysAdmin

Despite everything we were told by our parents, teachers, bosses, well-meaning authority figures, and hundreds of quotes about hard work that I found with a Google search, getting your work done well and on time is not the same as working hard. One does not necessarily imply the other.

I am a lazy SysAdmin. I am also a very productive SysAdmin. Those two seemingly contradictory statements are not mutually exclusive, rather they are complementary in a very positive way. Efficiency is the only way to make this possible.

This tenet is about working hard at the right tasks to optimize our own efficiency as SysAdmins. Part of this is about automation which we will explore in detail in Chapter 10 of Volume 2, but also throughout this course. The greater part of this tenet is about finding many of the myriad ways to use the short cuts already built into Linux.

Things like using aliases as shortcuts to reduce typing – but probably not in the way you think of them if you come from a Windows background. Naming files so that they can be easily found in lists, using the file name completion facility that is part of Bash, the default Linux shell for most distributions, and more, all contribute to making life easier for Lazy SysAdmins.

Automate Everything

The function of computers is to automate mundane tasks in order to allow us humans to concentrate on the tasks that the computers cannot – yet – do. For SysAdmins, those of us who run and manage the computers most closely, we have direct access to the tools that can help us work more efficiently. We should use those tools to maximum benefit.

In Chapter 8, “Be a Lazy SysAdmin”, of The Linux Philosophy for SysAdmins, I state, “A SysAdmin is most productive when thinking – thinking about how to solve existing problems and about how to avoid future problems; thinking about how to monitor Linux computers in order to find clues that anticipate and foreshadow those future problems; thinking about how to make her job more efficient; thinking about how to automate all of those tasks that need to be performed whether every day or once a year.”

SysAdmins are next most productive when creating the shell programs that automate the solutions that they have conceived while appearing to be unproductive. The more automation we have in place the more time we have available to fix real problems when they occur and to contemplate how to automate even more than we already have.

I have learned that, for me at least, writing shell programs – also known as scripts – provides the best single strategy for leveraging my time. Once having written a shell program it can be rerun as many times as needed.

Always use shell scripts

When writing programs to automate – well, everything – always use shell scripts rather than compiled utilities and tools. Because shell scripts are stored in ASCII plain text format, they can be easily viewed and modified by humans just as easily as they can by computers. You can examine a shell program and see exactly what it does and whether there are any obvious errors in the syntax or logic. This is a powerful example of what it means to be open.

A shell script or program is an executable file that contains at least one shell command. They usually have more than a single command and some shell scripts have thousands of lines of code. When taken together, these commands are the ones necessary to perform a desired task with a specifically defined result.

Context is important and this tenet should be considered in the context of our jobs as SysAdmins. The SysAdmin’s job differs significantly from those of developers and testers. In addition to resolving both hardware and software problems we manage the day to day operation of the systems under our care. We Monitor those systems for potential problems and make all possible efforts to prevent those problems before they impact our users. We install updates and perform full release level upgrades to the operating system. We resolve problems caused by our users. SysAdmins develop code to do all of those things and more; then we test that code; and then we support that code in a production environment.

Test early test often

“There is always one more bug.”

Lubarsky’s Law of Cybernetic Entomology

Lubarsky – whoever he might be – is correct. We can never find all of the bugs in our code. For every one I find there always seems to be another that crops up, usually at a very inopportune time.

Testing affects the ultimate outcome of the many tasks SysAdmins do and is an integral part of the Philosophy. However, testing is not just about programs. It is also about verification that problems – whether caused by hardware, software, or the seemingly endless ways that users can find to break things – that we are supposed to have resolved actually have been. These problems can be with application or utility software we wrote, system software, applications, and hardware. Just as importantly, testing is also about ensuring that the code is easy to use and the interface makes sense to the user.

Testing is hard work and it requires a well designed test plan based on the requirements statements. Regardless of the circumstances, start with a test plan. Even a very basic test plan provides some assurance that testing will be consistent and will cover the required functionality of the code.

Any good plan includes tests to verify that the code does everything it is supposed to. That is, if you enter X and click on button Y, you should get Z as the result. So you write a test that does creates those conditions and then verify that Z is the result.

The best plans include tests to determine how well the code fails. The specific scenarios explicitly covered by the test plan are important, but it they may fail to anticipate the havoc that can be caused by unanticipated or even completely random input. This situation can be at least partially covered by fuzzy testing in which someone or some tool randomly bangs on the keyboard until something bad happens.

For SysAdmins, testing in production, which some people consider to be a new thing, is a common practice. There is no test plan that can be devised by a lab full of testers that can possibly equal a few minutes in the real world of production.

Use common sense naming

The Lazy SysAdmin does everything possible to reduce unnecessarytyping and I take that seriously. This tenet expands on that but there is much more to it than just reducing the amount of typing I need to do. It is also about the readability of scripts and naming things so that they are more understandable more quickly.

One of the original Unix philosophy tenets was to always use lowercase and keep names short. An admirable goal but not one so easily met in the world of the SysAdmin. In many respects my own tenet would seem a complete refutation of the original.

However the original was intended for a different audience and this one is intended for SysAdmins with a different set of needs. Early Unix systems had very little memory compared to today’s systems so saving a few Bytes in a name was important. Unix and Linux are case sensitive so an extra keystroke to hit the shift key was extra work.

The ultimate goal is to create scripts that are readable and easily understood in order to make them easily maintainable. And then to use other, simple scripts and cron jobs to automate running those scripts. Keeping the script names reasonably short also reduces typing when executing those scripts from the command line but that is mostly irrelevant when starting them from another script or as cron jobs.

Readable scripts depend upon easily understandable and readable variable names. Sometimes, as with script names, these names may be longer but more understandable than many I have encountered in the past. Variable names like $DeviceName are much more understandable than $D5 and make a script easier to read.
Note that most of the Linux command names are short but they also have meaning. After working at the command line for a while you will understand most of these. For example the ls command means list the contents of a directory. Other command names contain the “ls” string in their names, such as lsusb to list the USB devices connected to the host or lsblk to list the block devices – hard drives – in the host.

Store data in open formats

The reason we use computers is to manipulate data. It used to be called “Data Processing” for a reason and that was an accurate description. We still process data although it may be in the form of video and audio streams, network and wireless streams, word processing data, spreadsheets, images, and more. It is all still just data.

We work with and manipulate text data streams with the tools we have available to us in Linux. That data usually needs to be stored and when there is a need to store data, it is always better to store it in open file formats than closed ones.

Although many user application programs store data in plain text formats including simple flat plain text and XML, this tenet is mostly about configuration data and scripts that relate directly to Linux. However, any type of data should be stored as plain text if possible.

“Open source” is about the code and making the source code available to any and all who want to view or modify it. “Open data” is about the openness of the data itself.

The term open data does not mean just having access to the data itself, it also means that the data can be viewed, used in some manner and shared with others. The exact manner in which those goals are achieved may be subject to some sort of attribution and open licensing. As with open source software, such licensing is intended to ensure the continued open availability of the data and not to restrict it any manner that would prevent its use.

Open data is knowable. That means that access to it is unfettered. Truly open data can be read freely and understood without the need for further interpretation or decryption. In the SysAdmin world, open means that the data we use to configure, monitor, and manage our Linux hosts is easy to find, read, and modify when necessary. It is stored in formats that permit that ease of access, such as plain text text. When a system is open the data and software can all be managed by open tools – tools that work with plain text text.

Use separate filesystems for data

There is a lot to this particular tenet and it requires understanding the nature of Linux filesystems and mount points.

Note: The primary meaning for the term “filesystem” in this tenet is a segment of the directory tree that is located on a separate partition or logical volume that must be mounted on a specified mount point of the root filesystem to enable access to it. We also use the term to describe the structure of the metadata on the partition or volume such as, EXT4, XFS, or other structure. These different usages should be clear from their context.

There are at least three excellent reasons for maintaining separate filesystems on our Linux hosts. First, when hard drives crash we may lose some or all of the data on a damaged filesystem but , as we will see, data on other filesystems on the crashed hard drive me still be salvageable.

Second, despite having access to huge amounts of hard drive space, it is possible to fill up a filesystem. When that happens separate filesystems can minimize the immediate effects and make recovery easier.

Third, upgrades can be made easier when certain filesystems such as /home are located on separate filesystems. This makes it easy to upgrade without needing to restore that data from a backup.

I have frequently encountered all three of these situations in my career. In some instances there was only a single partition, the root (/) partition, and so recovery was quite difficult. Recovery from these situations was always much easier and faster when the host was configured with separate filesystems.

Keeping data of all types safe is part of the SysAdmin’s job. Using separate filesystems for storing that data can help us accomplish that. This practice can also help us achieve our objective to be a Lazy Admin. Backups do allow us to recover most of the data that would otherwise be lost in a crash scenario, but using separate filesystem may allow us to recover all of the data from unaffected filesystems right up to the moment of a crash. Restoring from backups takes much longer.

Make programs portable

Portable programs make life much easier for the lazy SysAdmin. Portability is an important consideration because it allows programs to be used on a wide range of operating system and hardware platforms. Using interpretive languages such as Bash, Python, and Perl that can run on many types of systems can save loads of work.

Programs written in compiled languages such as C must be recompiled at the very least when porting from one platform to another. In many cases, platform specific code must be maintained in the sources in order to support the different hardware platforms that the binaries are expected to run on. This generates a lot of extra work, both writing and testing the programs.

Perl, Bash, and many other scripting languages are available in most environments. With very few exceptions programs written in Perl, Bash, Python, PHP, and other languages, can run unchanged on many different platforms.

Linux runs on a lot of hardware architectures14. Wikipedia maintains a long list of hardware architectures supported by Linux, but here are just a few. Of course Linux supports Intel and AMD. It also supports 32- and 64-bit ARM architectures that are found in practically every mobile phone on the planet and devices such as the Raspberry Pi15. Most mobile phones use a form of Linux called Android.

Use open source software

This tenet may not mean exactly what you think it does. Most times we think of open source software as something like the Linux kernel, LibreOffice, or any of the thousands of open source software packages that make up our favorite distribution. In the context of system administration, open source means the scripts that we write to automate our work.

Open source software is software with source code that anyone can inspect, modify, and enhance.

Opensource.com

The web page from which the quote above was taken contains a well-written discussion of open source software including some of the advantages of open source. I suggest you read that article and consider how it applies to the code we write – our scripts. The implications are there if we look for them.

The official definition of open source is quite terse. The annotated version of the open source definition17 at opensource.org contains ten sections that explicitly and succinctly define the conditions that must be be met for software to be considered truly open source. This definition is important to the Linux Philosophy for SysAdmins. You do not have to read this definition but I suggest you do so in order to gain a more complete understanding of what the term open source really means. However I can summarize a bit.

Open source software is open because it can be read, modified, and shared because its source code is freely available to anyone who wants it. This “free as in speech” approach to software promotes world-wide participation by individuals and organizations in the creation and testing of high quality code that can be shared freely by everyone. Being a good user of open source also means that we SysAdmins should share our own code, the code that we write to solve our own problems, and license it with one of the open source licenses.

Strive for elegance

Elegance is one of those things that can be difficult to define. I know it when I see it but putting what I see into a terse definition is a challenge. Using the Linux dict command, Wordnet provides one definition of elegance as, “a quality of neatness and ingenious simplicity in the solution of a problem (especially in science or mathematics); ‘the simplicity and elegance of his invention’”

In this context, I assert that elegance is a state of beauty and simplicity in the design and working of both hardware and software. When a design is elegant, software and hardware work better and are more efficient. The user is aided by simple, efficient, and understandable tools.

Creating elegance in a technological environment is hard. It is also necessary. Elegant solutions produce elegant results and are easy to maintain and fix. Elegance does not happen by accident; you must work for it.

Find the simplicity

The quality of simplicity is a large part of technical elegance. The tenets of the Linux Philosophy helped me to solidify my understanding of the truth that Linux is simple and that the simplicity is illuminated by the philosophy.

“UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity.”

Dennis Ritchie

In this tenet we search for the simplicity of Linux. I cringe when I see articles with titles like, “77 Linux commands and utilities you’ll actually use,” and “50 Most Frequently Used UNIX / Linux Commands (With Examples)”. These titles imply that there are sets of commands that you must memorize, or that knowing large numbers of commands is important.

I do read many of these articles, but I am usually looking for new and interesting commands; commands that might help me resolve a problem or simplify a command line program. I never tried to learn all of those Linux commands, regardless of what numbers you might come up with as the total for “all.”

I just started by learning the commands I needed at any given moment for whatever project was at hand. I started to learn more commands because I took on personal projects and ones for work that stretched my knowledge to the limit and forced me to find commands previously unknown to me in order to complete those projects. My repertoire of commands grew over time and I became more proficient at the application of those commands to resolve problems, I began finding jobs that paid me more and more money to play with Linux, my favorite toy.

As I learned about piping and redirection, about Standard Streams and STDIO, as I read about the Unix Philosophy and then the Linux Philosophy, I started to understand how and why the command line made Linux and the core utilities so powerful. I learned about the elegance of writing command line programs that manipulated data streams in amazing ways.

I also discovered that some commands are, if not completely obsolete, then seldom used and only in unusual circumstances. For this reason alone it does not make sense to find a list of Linux commands and memorize them. It is not an efficient use of your time as a SysAdmin to learn many commands that may never be needed. The simplicity here is to learn what you need to do the task at hand. There will be plenty of tasks in the future which will require you to learn other commands.

When writing our own administrative scripts, simplicity is also key. Each of our scripts should do only one thing and do it well. Complex programs are difficult to use and to maintain.

“Fools ignore complexity; pragmatists suffer it; experts avoid it; geniuses remove it.”

Alan Perlis

Use your favorite editor

Why is this a tenet of The Linux Philosophy for System Administrators? Because arguing about editors can be the cause of a great deal of wasted energy. Everyone has their favorite text editor and it might not be the same as mine. So what?

I use Vim as my editor. I have used it for years and like it very much. I am used to it. It meets my needs more than any other editor I have tried. If you can say that about your editor – whichever one that might be – then you are in editor nirvana.

I started using vi when I began learning Solaris over twenty years ago. My mentor suggested that I start learning to edit with vi because it would always be present on every system. That has proven to be true whether the operating system is Solaris or Linux. The vi editor is always there so I can count on it. For me, this works. Vim is the new vi but I can still use the vi command to launch Vim.

The vi editor can also used as the editor for Bash command line editing. Although the default for command editing is EMACS-like, I use the vi option because I already know the vi keystrokes. Other tools that use vi editing are the crontab and visudo commands; both of these are wrappers around vi. Lazy developers use code that already exists, especially when it is open source. Using existing editors for these tools is an excellent example of that.

It does not matter to me what tools you use and it should not matter to anyone else, either. What really matters is getting the job done. Whether you use vim or EMACS, systemd or SystemV, RPM or DEB, what difference does it make? The bottom line here is that you should use the tools with which you are most comfortable and that work best for you.

Document Everything

“Real programmers don’t comment their code, if it was hard to write, it should be hard to understand and harder to modify.”

unknown

I, too, would want to remain anonymous if I had written that. It might even have been meant to be sarcasm or irony. Regardless, this does seem to be the attitude of many developers and SysAdmins. There is a poorly disguised ethos amongst some developers and SysAdmins that one must figure everything out for themselves in order to join the club – whatever club that might be. If you cannot figure it out, they imply, you should go do something else because you don’t belong.

First, that is not true. Second, most developers, programmers, and SysAdmins that I know definitely do not subscribe to this view. In fact, the best ones, some of whom have been my mentors over the years, exemplify the exact opposite. The best of the best make documentation – good documentation – a high priority in everything they do.

I have used a lot of software whose creators subscribed to the philosophy that all code is self-explanatory. I have also been required to fix a lot of code that was completely uncommented and otherwise undocumented as well. It seems that many developers and SysAdmins figure if the program works for them it does not need to be documented. I have been the SysAdmin assigned to fix uncommented code on more than one occasion. That is one of the least enjoyable tasks I have ever had to do.

Part of the problem is that many PHBs do not see documentation as a high priority. I have been involved in many aspects of the IT industry and fortunately most of the companies I worked for believed that documentation was not only important, but that it was crucial to the task at hand, regardless of what that task was.

And yet there is a lot of really good documentation out there. For example the documentation for LibreOffice is excellent. It includes several documents in multiple formats including HTML and PDF that range from “Getting Started” to a very complete user’s guide for each of the LibreOffice applications.

The documentation for Red Hat Enterprise Linux (RHEL) and CentOS, and that for Fedora – which are all very closely related distributions – are also among the best I have seen in my more than 40 years of working in the IT industry.

Good documentation is not easy and takes time. It also requires an understanding of the audience – not only in relation to the purpose of the documentation, but also the technical expertise of the intended readers as well as the languages and cultures of the readers. Rich Bowen covered that quite nicely in his fine article at Opensource.com, “RTFM? How to write a manual worth reading”.

There is also the question of what constitutes good documentation for a SysAdmin. We explore these things in this tenet which is mostly about documenting the scripts we write.

Backup everything – frequently

Nothing can ever go wrong with my computer and I will never lose my data. If you believe that, I have a bridge you might like to buy.

I have experienced data loss for a myriad of reasons, many of them my own fault. Keeping decent backups has always enabled me to continue with minimal disruption. This tenet is concerned with some of the more common reasons for data loss, and methods for preventing data loss and facilitating easy recovery.

Recently, very recently, I encountered a problem in the form of a hard drive crash that destroyed the data in my home directory. I had been expecting this for some time so it came as no surprise. The first indication I had that something was wrong was a series of emails from the S.M.A.R.T (Self-Monitoring, Analysis and Reporting Technology) enabled hard drive on which my home directory resided23. Each of these emails indicated that one or more sectors had become defective and that the defective sectors had been taken off-line and reserved sectors allocated in their place. This is normal operation; hard drives are designed intentionally with reserved sectors for just this and the data is stored in a reserved sector instead of the intended one.

When the hard drive finally failed – I left it in my computer until it failed as a test – I replaced the drive, partitioned and formatted it appropriately, copied my files from the backup to the new drive, did a little testing, and was good to go.

Backups save time, effort, and money. Don’t be caught without backups. You will need them.

Follow your curiosity

People talk about life-long learning and how that keeps one mentally alert and youthful. The same is true of SysAdmins. There is always more to learn and I think that is what keeps most of us happy and always ready to tackle the next problem. Continuous learning helps to keep our minds and skills sharp no matter what or age.
I love to learn new things. I was fortunate in that my curiosity led me to a lifetime of working with my favorite toys – computers. There are certainly plenty of new things to learn about computers; the industry and technology are constantly changing. There are many things on Earth and in this Universe to be curious about. Computers and related technology just seem to be the thing I enjoy the most.

I also assume that you must be curious because you are reading this book. Curiosity got me into Linux in the first place but it was a long and winding road. Over a period of many years, my curiosity led me through many life events that led me to a job at IBM which led to writing the first training course for the original IBM PC, which led to a job at a company where I was able to learn Unix, which led me to Linux because Unix was too expensive to use at home, which led to a job at Red Hat which … you get the idea. Now I write about Linux.
Follow your own curiosity. You should explore the many aspects of Linux and go wherever your curiosity leads you. It was only by following my curiosity, first about electronics, then computers, programming, operating systems, Linux, servers, networking, and more, that I have been able to do so many fun and interesting things.

There is no should

This tenet is about possibilities. It is also the most Zen of all the tenets. It is more about how our minds work to solve problems than it is about specific technology. It is also about overcoming or at least recognizing some of the obstacles that prevent us from fully utilizing the potential we have in ourselves.

In “The Wrath of Kahn,” Spock says, “There are always possibilities.” With Linux there are always possibilities – many ways to approach and solve problems. This means that you may perform a task in one way while another SysAdmin may do it in another. There is no one way in which tasks “should” be done. There is only the way you have done it. If the results meet the requirements then the manner in which they were reached is perfection.
I believe that we Linux SysAdmins approach solving Linux problems with fewer contraints on our thinking than those who appear to think more in terms of “harnessing,” and “restrictions.” We have so many simple yet powerful tools available to us that we do not find ourselves constrained by either the operating system or any inhibitive manner of thinking about the tools we use or the operational methods with which we may apply them.

Rigid logic and rules do not give us SysAdmins enough flexibility to perform our jobs efficiently. We don’t especially care about how things “should” be done. SysAdmins are not easily limited by the “shoulds” that others try to constrain us with. We use logical and critical thinking that is flexible and that produces excellent resultsand which enables us to learn more while we are at it.

We don’t just think outside the box. We are the ones who destroy the boxes that others try to make us work inside. For us, there is no “should.”

Mentor the young SysAdmins

I have taken many training courses over the years and most have been very useful in helping me to learn more about Unix and Linux as well as a host of other subjects. But training – as useful and important as it is – cannot cover many essential aspects of performing SysAdmin duties. Some things can only be taught by a good mentor in a real world environment, usually while you are under extreme pressure to fix a critical problem. A good mentor will allow you to do the actual work in these situations so you can have a valuable learning experience while keeping the wolves at bay, taking the heat while you work uninterrupted. A great mentor will also be able to create a learning opportunity from every situation no matter how critical.

This tenet is also about teaching the young SysAdmins critical thinking and application of the scientific method to the art of solving problems. It is about passing on what you have received.

Support your favorite open source project

Linux and a very large proportion of the programs that we run on it are open source programs. Many of the larger projects, such as the kernel itself, are supported directly by foundations set up for that purpose, such as the Linux Foundation, and/or by corporations and other organizations that have an interest in doing so.

As a SysAdmin I write a lot of scripts and I like doing so but I am not an application programmer. Nor do I want to be because I enjoy the work of a SysAdmin which allows for a different kind of programming. So, for the most part, contributing code to an open source project is not a good option for me. There are other ways to contribute, such as answering questions on lists or web sites, submitting bug reports, writing documentation, writing articles for web sites like Opensource.com, teaching, and contributing money. And I use some of those options. This tenet is about exploring some of the ways in which you might contribute. As in mentoring, this is a way to give back to the community.

Reality Bytes

The Linux Philosophy for SysAdmins is a technical philosophy which would not normally be considered to be very practical. But there is “truth” here. Reality imposes itself upon SysAdmins every day in a multitude of ways. It is possible always to be able to follow each of the tenets – but it is quite improbable. In the “real” world, we SysAdmins face some incredible challenges just to get our assigned work completed. Deadlines, management, and other pressures force us to make decisions many times a day about what to do next and how to do it. Meetings usually waste our time – not always but usually. Finding time and money for training is unheard of in many organizations and requires selling your SysAdmin soul in others.

Adhering to the Philosophy does pay high-value returns in the long run. Still, reality always intrudes on the ever so perfect philosophical realm. Without room for flexibility any philosophy is merely doctrine and that is not what the Linux Philosophy for System Administrators is about. This tenet explores how reality may affect us as System Administrators.

Computers are easy – people are hard

Bridget Kromhout

SysAdmins must work and interact with people. It can be a difficult but we do need to do so from time to time. We SysAdmins must interact with other people whether they be users, technical professionals on other teams, our peers, or management. We need to discuss our work with other people who have differing levels of knowledge. Knowledge is not a binary condition it is analog. People have a wide disparity in the amount of knowledge that they have about computers and technology. This ranges from seemingly less than none to very knowledgeable. Their level of knowledge is important in how we interact with them.

I have been accused of over-explaining things but I would rather over-explain than under-explain. Some have even called it “mansplaining,” but that is not really my intent. I have found that all technical people regardless of gender, gender preference or identification, or other identifying characteristic, we all have the same tendency to explain things from the ground up when asked a simple question. That is because the answers are never as simple as the questions.

Summary

This article is a brief overview of my book, “The Linux Philosophy for SysAdmins.” The Philosophy is my mental framework for how I work as a SysAdmin. It has been a useful tool for me. Working in accordance with the tenets of the Philosophy has enhanced my productivity and efficiency.