Shell Scripting

How to Create a Shell Script in Linux

Suparna Ganguly — Wed, 28 Jul 2021 16:00:00 +0000

Do you want to create a Shell script in your Linux system?

This guide will take you through how to create a shell script using multiple text editors, how to add comments, and how to use Shell variables.

But before heading over to creating a shell script, let’s understand what Shell scripting in Linux is.

What is Shell Scripting in Linux?

So, what’s Shell scripting?

Shell Scripting is defined as an open-source program that’s run by Linux or Unix shell. Through shell scripting, you can write commands to be executed by the shell.

Lengthy and repetitive commands are usually combined into a simple command script. You can store this script and execute it whenever needed.

Shell scripting in Linux makes programming effortless.

Ways of Creating a Simple Shell Script in Linux

Creating a simple shell script in Linux is very easy. You can do that using multiple text editors. This tutorial will show how to create a shell script with two different methods, such as 1) using the default text editor, and 2) Using the Vim text editor tool.

Method 1: Using the Default Text Editor

To create a shell script using the default text editor, just follow the steps given below.

Step 1: Create a text file having a “.sh” extension. Then type a simple script.

Step 2: Now don’t change the directory. And open the terminal. Using the command below, give executable access to the file created.

chmod +x testing.sh

Step 3: Execute the below-given script in the terminal:

./testing.sh

This was a simple technique of creating a shell script in Linux using the default editor. Now, let’s look at the next method.

Method 2: Using the Vim Text Editor Tool

Vim text editor tool is a tool that helps create a shell script in a Linux system. In case you don’t have it already installed, use the command to install Vim:

sudo apt install vim

Now follow the steps for creating a shell script using the tool.

Step 1: For opening the editor, simply type:

vim

Step 2: Once you’re in, open the terminal. Then create a bash file via:

vi testing.sh

After the execution of the command, the editor will appear as below.

Go to Full Article

How to Add a Simple Progress Bar in Shell Script

Nawaz Abbasi — Mon, 26 Oct 2020 18:40:25 +0000

by Nawaz Abbasi

At times, we need to write shell scripts that are interactive and user executing them need to monitor the progress. For such requirements, we can implement a simple progress bar that gives an idea about how much task has been completed by the script or how much the script has executed.

To implement it, we only need to use the “echo” command with the following options and a backslash-escaped character.

-n : do not append a newline
-e : enable interpretation of backslash escapes
\r : carriage return (go back to the beginning of the line without printing a newline)

For the sake of understanding, we will use “sleep 2” command to represent an ongoing task or a step in our shell script. In a real scenario, this could be anything like downloading files, creating backup, validating user input, etc. Also, to give an example we are assuming only four steps in our script below which is why we are using 20,40,60,80 (%) as progress indicator. This can be adjusted as per the number of steps in a script. For instance, a script with three steps can be represented by 33,66,99 (%) or a script with ten steps can be represented by 10-90 (%) as progress indicator.

The implementation looks like the following:

echo -ne '>>>                       [20%]\r'
# some task
sleep 2
echo -ne '>>>>>>>                   [40%]\r'
# some task
sleep 2
echo -ne '>>>>>>>>>>>>>>            [60%]\r'
# some task
sleep 2
echo -ne '>>>>>>>>>>>>>>>>>>>>>>>   [80%]\r'
# some task
sleep 2
echo -ne '>>>>>>>>>>>>>>>>>>>>>>>>>>[100%]\r'
echo -ne '\n'

In effect, every time the “echo” command executes, it replaces the output of the previous “echo” command in the terminal thus representing a simple progress bar. The last “echo” command simply enters a newline (\n) in the terminal to resume the prompt for the user.

The execution looks like the following:

Go to Full Article

Quick Tutorial on How to Use Shell Scripting in Linux: Coin Toss App

Nawaz Abbasi — Mon, 12 Oct 2020 19:08:19 +0000

by Nawaz Abbasi

Simply put, a Shell Script is a program that is run by a UNIX/Linux shell. It is a file that contains a series of commands which are executed sequentially as if they were entered on the command line interface (CLI) or terminal.

In this quick tutorial on Shell Scripting, we will write a simple program to toss a coin. Basically, the output of our program should be either HEADS or TAILS (of course, randomly).

To start with, the first line of a shell script should indicate which interpreter/shell is to be used to execute the script. In this tutorial we will be using /bin/bash and it will be denoted as #!/bin/bash which is called a shebang!

Next, we will be using an internal Bash function - a shell variable named RANDOM. It returns a random (actually, pseudorandom) integer in the range 0-32767. We will use this variable to get 2 random values – either 0 (for HEADS) or 1 (for TAILS). This will be done via a simple arithmetic operation in shell using % (Modulus operator, returns remainder), $((RANDOM%2)) and this will be stored in a result variable. So, the second line of our program becomes Result=$((RANDOM%2)) – Note that there should be no space around = (assignment operator) while assigning value to a variable in shell scripts.

At last, we just need to print HEADS if we got 0 or TAILS if we got 1, in the Result variable. Perhaps you guessed it by now, we will use if conditional statements for this. Within the conditions, we will compare the value of Result variable with 0 and 1; and print HEADS or TAILS accordingly. For this, the operator for integer comparison -eq (is equal to) is used to check if the value of two operands are equal or not.

Ergo, our shell script looks like the following:

#!/bin/bash
Result=$((RANDOM%2))
if [[ ${Result} -eq 0 ]]; then
    echo HEADS
elif [[ ${Result} -eq 1 ]]; then
    echo TAILS
fi

Let’s say we name the script cointoss.sh – Note that .sh is only to make it identifiable for user(s) that the file/script is a shell script. And, Linux is an Extensionless system.

Finally, to run the script we need to make it executable and that can be done by using the chmod command – chmod +x cointoss.sh

Few script executions:

$ ./cointoss.sh

TAILS

$ ./cointoss.sh

HEADS

$ ./cointoss.sh

HEADS

$ ./cointoss.sh

TAILS

To wrap up, in this quick tutorial about writing shell scripts, we learned about shebang, RANDOM, variable assignment, an arithmetic operation using Modulus operator %, if conditional statements, integer comparison operator -eq and executing a shell script.

Go to Full Article

Bash Shell Games: Let's Play Go Fish!

Dave Taylor — Tue, 30 Jul 2019 12:00:00 +0000

by Dave Taylor

How to begin developing a computer version of the popular card game.

Between the previous 163 columns I've written here in Linux Journal and the dozens of games I programmed and explored during the creation of my Wicked Cool Shell Scripts book, I've written a lot of Bash shell games. The challenge is to find one that's simple enough where a shell script will work, but isn't so simple that it ends up being only a half-dozen lines.

Magic 8-Ball is a perfect example. It turns out that the entire "predict the future" gizmo was really just a 20-sided die floating in dark purple fluid. So an array of 20 possible values and a random number selector and boom—you've got a magic 8-ball script:


#!/bin/sh

# magic 8 ball. Yup. Pick a random number, output message

# messages harvested from the Wikipedia entry

answers=("It is certain." "It is decidedly so."
  "Without a doubt." "Yes - definitely."
  "You may rely on it." "As I see it, yes." "Most likely."
  "Outlook good." "Yes." "Signs point to yes."
  "Reply hazy, try again." "Ask again later."
  "Better not tell you now." "Cannot predict now."
  "Concentrate and ask again." "Don't count on it."
  "My reply is no." "My sources say no."
  "Outlook not so good." "Very doubtful.")

echo "Oh! Magic 8 Ball, Please Tell Me True..." ; echo ""
/bin/echo -n "What is your question? "
read question

answer=$(( $RANDOM % 20 ))

echo ""
echo "I have looked into the future and I say: "
echo "     ${answers[$answer]}" ; echo ""

exit 0

Let's do a quick run to see if I'm the most popular LJ writer:


$ sh magic8.sh
Oh! Magic 8 Ball, Please Tell Me True...

What is your question? Am I the most popular LJ writer?

I have looked into the future and I say:
     My reply is no.

Ouch, that's harsh. I write the darn divination program, and it just drops a brick on my foot. Yeesh.

More seriously, Magic 8 Ball is too simple to make an interesting shell script. By contrast, Call of Duty is way too complex, even if I did a version with text output instead of gorgeously rendered 3D graphics.

Card Game Function Library

That's why card games prove to be good as programming challenges or exercises: the core mechanism of a 52-card random deck is pretty straightforward, so it's all about the actual cardplay.

Not only that, but as I've written before about card games as shell scripts, I already have a handy set of functions to create, shuffle and display cards out of a deck. If you want to rummage in the archives, I've tackled Acey-Deucey, Baccarat and some bits and pieces of Cribbage.

In order to jump right into the new game that I'm going to describe how to build, Go Fish!, let's steal the following functions from my earlier scripts:

Go to Full Article

Finishing Up the Bash Mail Merge Script

Dave Taylor — Thu, 04 Jul 2019 12:00:00 +0000

by Dave Taylor

Finally, I'm going to finish the mail merge script, just in time for Replicant Day.

Remember the mail merge script I started writing a while back? Yeah, that was quite some time ago. I got sidetracked with the Linux Journal Anniversary special issue (see my article "Back in the Day: UNIX, Minix and Linux"), and then I spun off on a completely different tangent for my last article ("Breaking Up Apache Log Files for Analysis"). I blame it on...

SQUIRREL!

Oh, sorry, back to topic here. I was developing a shell script that would let you specify a text document with embedded field names that could be substituted iteratively across a file containing lots of field values.

Each field was denoted by #fieldname#, and I identified two categories of fieldnames: fixed and dynamic. A fixed value might be #name#, which would come directly out of the data file, while a dynamic value could be #date#, which would be the current date.

More interesting, I also proposed calculated values, specifically #suggested#, which would be a value calculated based on #donation#, and #date#, which would be replaced by the current date. The super-fancy version would have a simple language where you could define the relationship between variables, but let's get real. Mail merge. It's just mail merge.

Reading and Assigning Values

It turns out that the additions needed for this script aren't too difficult. The basic data file has comma-separated field names, then subsequent lines have the values associated with those fields.

Here's that core code:


if [ $lines -eq 1 ] ; then # field names
# grab variable names
declare -a varname=($f1 $f2 $f3 $f4 $f5 $f6 $f7)
else # process fields

# grab values for this line (can contain spaces)
declare -a value=("$f1" "$f2" "$f3" "$f4" "$f5" "$f6" "$f7")

The declare function turns out to be ideal for this, allowing you to create an array varname based on the contents of the first line, then keep replacing the values of the array value, so that varname[1] = value[1], and so on.

To add the additional variables #date# and #suggested#, you simply can append them to the varname and value arrays. The first one is easy, but it did highlight a weakness in the original code that I had to fix by adding quotes as shown:

Go to Full Article

Breaking Up Apache Log Files for Analysis

Dave Taylor — Mon, 27 May 2019 11:00:00 +0000

by Dave Taylor

Dave tackles analysis of the ugly Apache web server log.

I know, in my last article I promised I'd jump back into the mail merge program I started building a while back. Since I'm having some hiccups with my AskDaveTaylor.com web server, however, I'm going to claim editorial privilege and bump that yet again.

What I need to do is be able to process Apache log files and isolate specific problems and glitches that are being encountered—a perfect use for a shell script. In fact, I have a script of this nature that offers basic analytics in my book Wicked Cool Shell Scripts from O'Reilly, but this is a bit more specific.

Oh Those Ugly Log Files

To start, let's take a glance at a few lines out of the latest log file for the site:


$ head sslaccesslog_askdavetaylor.com_3_8_2019
18.144.59.52 - - [08/Mar/2019:06:10:09 -0600] "GET /wp-content/
↪themes/jumpstart/framework/assets/js/nivo.min.js?ver=3.2
 ↪HTTP/1.1" 200 3074
"https://www.askdavetaylor.com/how-to-play-dvd-free-windows-
↪10-win10/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
 ↪AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
 ↪64.0.3282.140 Safari/537.36 Edge/18.17763 X-Middleton/1"
 ↪52.53.151.37 - - [08/Mar/2019:06:10:09 -0600] "GET
 ↪/wp-includes/js/jquery/jquery.js?ver=1.12.4 HTTP/1.1"
 ↪200 33766 "https://www.askdavetaylor.com/how-to-play
↪-dvd-free-windows-10-win10/" "Mozilla/5.0 (Windows NT
 ↪10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
 ↪Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763
 ↪X-Middleton/1" 18.144.59.52 - - [08/Mar/2019:06:10:09
 ↪-0600] "GET /wp-content/plugins/google-analytics-for-
↪wordpress/assets/js/frontend.min.js?ver=7.4.2 HTTP/1.1"
 ↪200 2544 "https://www.askdavetaylor.com/how-to-play
↪-dvd-free-windows-10-win10/"
 ↪"Mozilla/5.0 (Windows NT 10.0; Win64; x64)
 ↪AppleWebKit/537.36 (KHTML, like Gecko)
 ↪Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763
 ↪X-Middleton/1"

It's big and ugly, right? Okay, then let's just isolate a single entry to see how it's structured:


18.144.59.52 - - [08/Mar/2019:06:10:09 -0600] "GET
 ↪/wp-content/themes/jumpstart/framework/assets/js/
↪nivo.min.js?ver=3.2 HTTP/1.1" 200 3074
"https://www.askdavetaylor.com/how-to-play-dvd-free-windows-
↪10-win10/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140
 ↪Safari/537.36 Edge/18.17763 X-Middleton/1"

That's still obfuscated enough to kick off a migraine!

Fortunately, the Apache website has a somewhat clearer explanation of what's known as the custom log file format that's in use on my server. Of course, it's described in a way that only a programmer could love:

Go to Full Article

What The @#$%&! (Heck) is this #! (Hash-Bang) Thingy In My Bash Script

Mitch Frazier — Fri, 10 May 2019 11:30:00 +0000

by Mitch Frazier

You've seen it a million times—the hash-bang (#!) line at the top of a script—whether it be Bash, Python, Perl or some other scripting language. And, I'm sure you know what its purpose is: it specifies the script interpreter that's used to execute the script. But, do you know how it actually works? Your initial thought might be that your shell (bash) reads that line and then executes the specified interpreter, but that's not at all how it works. How it actually works is the main focus of this post, but I also want to introduce how you can create your own version of "hash-bang" if you're so inclined.

Go to Full Article

Fun with Mail Merge and Cool Bash Arrays

Dave Taylor — Mon, 25 Mar 2019 11:30:00 +0000

by Dave Taylor

Creating a sed-based file substitution tool.

A few weeks ago, I was digging through my spam folder and found an email message that started out like this:


Dear #name#
Congratulations on winning the $15.7 million lottery payout!
To learn how to claim your winnings, please...

Obviously, it was a scam (does anyone actually fall for these?), but what captured my attention was the #name# sequence. Clearly that was a fail on the part of the sender who presumably didn't know how to use AnnoyingSpamTool 1.3 or whatever the heck he or she was using.

The more general notation for bulk email and file transformations is pretty interesting, however. There are plenty of legitimate reasons to use this sort of substitution, ranging from email newsletters (like the one I send every week from AskDaveTaylor.com—check it out!) to stockholder announcements and much more.

With that as the inspiration, let's build a tool that offers just this capability.

The simple version will be a 1:1 substitution, so #name# becomes, say, "Rick Deckard", while #first# might be "Rick" and #last# might be "Deckard". Let's build on that, but let's start small.

Simple Word Substitution in Linux

There are plenty of ways to tackle the word substitution from the command line, ranging from Perl to awk, but here I'm using the original UNIX command sed (stream editor) designed for exactly this purpose. General notation for a substitution is s/old/new/, and if you tack on a g at the end, it matches every occurrence on a line, not only the first, so the full command is s/old/new/g.

Before going further, here's a simple document that has necessary substitutions embedded:


$ cat convertme.txt
#date#

Dear #name#, I wanted to start by again thanking you for your
generous donation of #amount# in #month#. We couldn't do our
work without support from humans like you, #first#.

This year we're looking at some unexpected expenses,
particularly in Sector 5, which encompasses #state#, as you
know. I'm hoping you can start the year with an additional
contribution? Even #suggested# would be tremendously helpful.

Thanks for your ongoing support. With regards,

Rick Deckard
Society for the Prevention of Cruelty to Replicants

Scan through it, and you'll see there's a lot of substitutions to do: #date#, #name#, #amount#, #month#, #first#, #state# and #suggested#. It turns out that #date# will be replaced with the current date, and #suggested# is one that'll be calculated as the letter is processed, but that's for a bit later, so stay tuned for that.

Go to Full Article

Converting Decimals to Roman Numerals with Bash

Dave Taylor — Mon, 18 Feb 2019 12:30:00 +0000

by Dave Taylor

Decimals to Roman numerals—here we hit all the limitations of Bash shell scripting.

My last few articles have given me a chance to relive my undergraduate computer science degree and code a Roman numeral to decimal converter. It's quite handy when you're watching old movies (when was MCMLVII anyway?), and the basic coding algorithm was reasonably straightforward. (See Dave's "Roman Numerals and Bash" and "More Roman Numerals and Bash".)

The trick with Roman numerals, however, is that it's what's known as a subtractive notation. In other words, it's not a position → value or even symbol → value notation, but a sort of hybrid. MM = 2000, and C = 100, but MMC and MCM are quite different: the former is 2100, and the latter is 1000 + (–100 + 1000) = 1900.

This means that the conversion isn't quite as simple as a mapping table, which makes it a good homework assignment for young comp-sci students!

Let's Write Some Code

In the Roman numeral to decimal conversion, a lot of the key work was done by this simple function:


mapit() {
   case $1 in
     I|i) value=1 ;;
     V|v) value=5 ;;
     X|x) value=10 ;;
     L|l) value=50 ;;
     C|c) value=100 ;;
     D|d) value=500 ;;
     M|m) value=1000 ;;
      * ) echo "Error: Value $1 unknown" >&2 ; exit 2 ;;
   esac
}

You'll need this function to proceed, but as a cascading set of conditional statements. Indeed, in its simple form, you could code a decimal to Roman numeral converter like this:


while [ $decvalue -gt 0 ] ; do

  if [ $decvalue -gt 1000 ] ; then
    romanvalue="$romanvalue M"
    decvalue=$(( $decvalue - 1000 ))
  elif [ $decvalue -gt 500 ] ; then
    romanvalue="$romanvalue D"
    decvalue=$(( $decvalue - 500 ))
  elif [ $decvalue -gt 100 ] ; then
    romanvalue="$romanvalue C"
    decvalue=$(( $decvalue - 100 ))
  elif [ $decvalue -gt 50 ] ; then
    romanvalue="$romanvalue L"
    decvalue=$(( $decvalue - 50 ))
  elif [ $decvalue -gt 10 ] ; then
    romanvalue="$romanvalue X"
    decvalue=$(( $decvalue - 10 ))
  elif [ $decvalue -gt 5 ] ; then
    romanvalue="$romanvalue V"
    decvalue=$(( $decvalue - 5 ))
  elif [ $decvalue -ge 1 ] ; then
    romanvalue="$romanvalue I"
    decvalue=$(( $decvalue - 1 ))
  fi

done

This actually works, though the results are, um, a bit clunky:


$ sh 2roman.sh 25
converts to roman numeral  X X I I I I I

Or, more overwhelming:

Go to Full Article

Writing Secure Shell Scripts

Dave Taylor — Tue, 05 Feb 2019 12:30:00 +0000

by Dave Taylor

Don't expose your system with sloppy scripts!

Although a Linux desktop or server is less susceptible to viruses and malware than a typical Windows device, there isn't a device on the internet that isn't eventually attacked. The culprit might be the stereotypical nerd in a bedroom testing his or her hacker chops (think Matthew Broderick in War Games or Angelina Jolie in Hackers). Then again, it might be an organized military, criminal, terrorist or other funded entity creating massive botnets or stealing millions of credit cards via a dozen redirected attack vectors.

In any case, modern systems face threats that were unimaginable in the early days of UNIX development and even in the first few years of Linux as a hobbyist reimplementation of UNIX. Ah, back in the day, the great worry was about copyrighted code, and so useful tools constantly were being re-implemented from scratch to get away from the AT&T Bell Labs licenses and so forth.

I have personal experience with this too. I rewrote the Hunt the Wumpus game wumpus from scratch for BSD 4.2 when the Berkeley crowd was trying to get away from AT&T UNIX legal hassles. I know, that's not the greatest claim to fame, but I also managed to cobble together a few other utilities in my time too.

Evolution worked backward with the internet, however. In real life, the lawless Wild West was gradually tamed, and law-abiding citizens replaced the outlaws and thugs of the 1850s and the Gold Rush. Online, it seems that there are more, smarter and better organized digital outlaws than ever.

Which is why one of the most important steps in learning how to write shell scripts is to learn how to ensure that your scripts are secure—even if it's just your own home computer and an old PC you've converted into a Linux-based media server with Plex or similar.

Let's have a look at some of the basics.

Know the Utilities You Invoke

Here's a classic trojan horse attack: an attacker drops a script called ls into /tmp, and it simply checks to see the userid that invoked it, then hands off its entire argument sequence to the real /bin/ls. If it recognizes userid = root, it makes a copy of /bin/sh into /tmp with an innocuous name, then changes its permission to setuid root.

This is super easy to write. Here's a version off the top of my head:


#!/bin/sh

if [ "$USER" = "root" ] ; then
  /bin/cp /bin/sh /tmp/.secretshell
  /bin/chown root /tmp/.secretshell
  /bin/chmod 4666 root /tmp/.secretshell
fi

exec /bin/ls $*

I hope you understand what just happened. This simple little script has created a shell that always grants its user root access to the Linux system. Yikes. Fancier versions would remove themselves once the root shell has been created, leaving no trace of how this transpired.

Go to Full Article