Progamming

Getting Started with Rust: Working with Files and Doing File I/O

Mihalis Tsoukalos — Thu, 20 Jun 2019 11:30:00 +0000

How to develop command-line utilities in Rust.

This article demonstrates how to perform basic file and file I/O operations in Rust, and also introduces Rust's ownership concept and the Cargo tool. If you are seeing Rust code for the first time, this article should provide a pretty good idea of how Rust deals with files and file I/O, and if you've used Rust before, you still will appreciate the code examples in this article.

Ownership

It would be unfair to start talking about Rust without first discussing ownership. Ownership is the Rust way of the developer having control over the lifetime of a variable and the language in order to be safe. Ownership means that the passing of a variable also passes the ownership of the value to the new variable.

Another Rust feature related to ownership is borrowing. Borrowing is about taking control over a variable for a while and then returning that ownership of the variable back. Although borrowing allows you to have multiple references to a variable, only one reference can be mutable at any given time.

Instead of continuing to talk theoretically about ownership and borrowing, let's look at a code example called ownership.rs:


fn main() {
    // Part 1
    let integer = 321;
    let mut _my_integer = integer;
    println!("integer is {}", integer);
    println!("_my_integer is {}", _my_integer);
    _my_integer = 124;
    println!("_my_integer is {}", _my_integer);

    // Part 2
    let a_vector = vec![1, 2, 3, 4, 5];
    let ref _a_correct_vector = a_vector;
    println!("_a_correct_vector is {:?}", _a_correct_vector);

    // Part 3
    let mut a_var = 3.14;
    {
        let b_var = &mut a_var;
        *b_var = 3.14159;
    }
    println!("a_var is now {}", a_var);
}

So, what's happening here? In the first part, you define an integer variable (integer) and create a mutable variable based on integer. Rust performs a full copy for primitive data types because they are cheaper, so in this case, the integer and _my_integer variables are independent from each other.

However, for other types, such as a vector, you aren't allowed to change a variable after you have assigned it to another variable. Additionally, you should use a reference for the _a_correct_vector variable of Part 2 in the above example, because Rust won't make a copy of a_vector.

Go to Full Article

Breaking Up Apache Log Files for Analysis

Dave Taylor — Mon, 27 May 2019 11:00:00 +0000

by Dave Taylor

Dave tackles analysis of the ugly Apache web server log.

I know, in my last article I promised I'd jump back into the mail merge program I started building a while back. Since I'm having some hiccups with my AskDaveTaylor.com web server, however, I'm going to claim editorial privilege and bump that yet again.

What I need to do is be able to process Apache log files and isolate specific problems and glitches that are being encountered—a perfect use for a shell script. In fact, I have a script of this nature that offers basic analytics in my book Wicked Cool Shell Scripts from O'Reilly, but this is a bit more specific.

Oh Those Ugly Log Files

To start, let's take a glance at a few lines out of the latest log file for the site:


$ head sslaccesslog_askdavetaylor.com_3_8_2019
18.144.59.52 - - [08/Mar/2019:06:10:09 -0600] "GET /wp-content/
↪themes/jumpstart/framework/assets/js/nivo.min.js?ver=3.2
 ↪HTTP/1.1" 200 3074
"https://www.askdavetaylor.com/how-to-play-dvd-free-windows-
↪10-win10/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
 ↪AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
 ↪64.0.3282.140 Safari/537.36 Edge/18.17763 X-Middleton/1"
 ↪52.53.151.37 - - [08/Mar/2019:06:10:09 -0600] "GET
 ↪/wp-includes/js/jquery/jquery.js?ver=1.12.4 HTTP/1.1"
 ↪200 33766 "https://www.askdavetaylor.com/how-to-play
↪-dvd-free-windows-10-win10/" "Mozilla/5.0 (Windows NT
 ↪10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
 ↪Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763
 ↪X-Middleton/1" 18.144.59.52 - - [08/Mar/2019:06:10:09
 ↪-0600] "GET /wp-content/plugins/google-analytics-for-
↪wordpress/assets/js/frontend.min.js?ver=7.4.2 HTTP/1.1"
 ↪200 2544 "https://www.askdavetaylor.com/how-to-play
↪-dvd-free-windows-10-win10/"
 ↪"Mozilla/5.0 (Windows NT 10.0; Win64; x64)
 ↪AppleWebKit/537.36 (KHTML, like Gecko)
 ↪Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763
 ↪X-Middleton/1"

It's big and ugly, right? Okay, then let's just isolate a single entry to see how it's structured:


18.144.59.52 - - [08/Mar/2019:06:10:09 -0600] "GET
 ↪/wp-content/themes/jumpstart/framework/assets/js/
↪nivo.min.js?ver=3.2 HTTP/1.1" 200 3074
"https://www.askdavetaylor.com/how-to-play-dvd-free-windows-
↪10-win10/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140
 ↪Safari/537.36 Edge/18.17763 X-Middleton/1"

That's still obfuscated enough to kick off a migraine!

Fortunately, the Apache website has a somewhat clearer explanation of what's known as the custom log file format that's in use on my server. Of course, it's described in a way that only a programmer could love:

Go to Full Article

Creating Linux Command-Line Tools in Clojure

Mihalis Tsoukalos — Fri, 29 Mar 2019 12:00:00 +0000

by Mihalis Tsoukalos

Learn how the leiningen utility can help you manage your Clojure projects.

This article is a gentle introduction to the Clojure Functional Programming language that is based on LISP, uses the Java JVM and has a handy REPL. And, as Clojure is based on LISP, be prepared to see lots of parentheses!

Installing Clojure

You can install Clojure on a Debian Linux machine by executing the following command as root or using sudo:


# apt-get install clojure

Finding the version of Clojure you are using is as simple as executing one of the following commands inside the Clojure REPL, which you can enter by running clojure:


# clojure
Clojure 1.8.0
user=> *clojure-version*
{:major 1, :minor 8, :incremental 0, :qualifier nil}
user=> (clojure-version)
"1.8.0"
user=> (println *clojure-version*)
{:major 1, :minor 8, :incremental 0, :qualifier nil}
nil

The first command gets you into the Clojure REPL, which displays the user=> prompt and waits for user input. The remaining three commands that should be executed within the Clojure REPL will generate the same output, which, in this example, shows that Clojure version 1.8.0 is being used. So, if you're following along, congratulations! You have just run your first Clojure code!

The leiningen Utility

The first thing you should do after getting Clojure is to install a very handy utility named leiningen, which is the easiest way to use and manage Clojure projects on your Linux machine. Follow the instructions at leiningen.org or use your favourite package manager to install leiningen on your Linux machine. Additionally, if you are using Clojure all the time and working with large Clojure projects, tools like Jenkins and Semaphore will automate your build and test phases and save you lots of time.

After installing leiningen, use the lein command (which is the name of the executable file for the leiningen package) to create a new project named hw:


$ lein new hw
Generating a project called hw based on the 'default' template.
The default template is intended for library projects,
not applications. To see other templates (app, plugin, etc),
try `lein help new`.

The preceding command will create a new directory named hw that will contain files and other directories. You'll need to make some changes to some of the project files in order to execute the project. First, you'll need to edit the project.clj that can be found inside the hw directory and make it as follows:

Go to Full Article

Easier Python paths with pathlib

Reuven M. Lerner — Mon, 11 Feb 2019 12:30:00 +0000

by Reuven M. Lerner

A look at the benefits of using pathlib, the "object-oriented way of dealing with paths".

Working with files is one of the most common things developers do. After all, you often want to read from files (to read information saved by other users, sessions or programs) or write to files (to record data for other users, sessions or programs).

Of course, files are located inside directories. Navigating through directories, finding files in those directories, and even extracting information about directories (and the files within them) might be common, but they're often frustrating to deal with. In Python, a number of different modules and objects provide such functionality, including os.path, os.stat and glob.

This isn't necessarily bad; the fact is that Python developers have used this combination of modules, methods and files for quite some time. But if you ever felt like it was a bit clunky or old-fashioned, you're not alone.

Indeed, it turns out that for several years already, Python's standard library has come with the pathlib module, which makes it easier to work with directories and files. I say "it turns out", because although I might be a long-time developer and instructor, I discovered "pathlib" only in the past few months—and I must admit, I'm completely smitten.

pathlib has been described as an object-oriented way of dealing with paths, and this description seems quite apt to me. Rather than working with strings, instead you work with "Path" objects, which not only allows you to use all of your favorite path- and file-related functionality as methods, but it also allows you to paper over the differences between operating systems.

So in this article, I take a look at pathlib, comparing the ways you might have done things before to how pathlib allows you to do them now.

pathlib Basics

If you want to work with pathlib, you'll need to load it into your Python session. You should start with:


import pathlib

Note that if you plan to use certain names from within pathlib on a regular basis, you'll probably want to use from-import. However, I strongly recommend against saying from pathlib import *, which will indeed have the benefit of importing all of the module's names into the current namespace, but it'll also have the negative effect of importing all of the module's names into the current namespace. In short, import only what you need.

Now that you've done that, you can create a new Path object. This allows you to represent a file or directory. You can create it with a string, just as you might do a path (or filename) in more traditional Python code:

Go to Full Article

Writing Secure Shell Scripts

Dave Taylor — Tue, 05 Feb 2019 12:30:00 +0000

by Dave Taylor

Don't expose your system with sloppy scripts!

Although a Linux desktop or server is less susceptible to viruses and malware than a typical Windows device, there isn't a device on the internet that isn't eventually attacked. The culprit might be the stereotypical nerd in a bedroom testing his or her hacker chops (think Matthew Broderick in War Games or Angelina Jolie in Hackers). Then again, it might be an organized military, criminal, terrorist or other funded entity creating massive botnets or stealing millions of credit cards via a dozen redirected attack vectors.

In any case, modern systems face threats that were unimaginable in the early days of UNIX development and even in the first few years of Linux as a hobbyist reimplementation of UNIX. Ah, back in the day, the great worry was about copyrighted code, and so useful tools constantly were being re-implemented from scratch to get away from the AT&T Bell Labs licenses and so forth.

I have personal experience with this too. I rewrote the Hunt the Wumpus game wumpus from scratch for BSD 4.2 when the Berkeley crowd was trying to get away from AT&T UNIX legal hassles. I know, that's not the greatest claim to fame, but I also managed to cobble together a few other utilities in my time too.

Evolution worked backward with the internet, however. In real life, the lawless Wild West was gradually tamed, and law-abiding citizens replaced the outlaws and thugs of the 1850s and the Gold Rush. Online, it seems that there are more, smarter and better organized digital outlaws than ever.

Which is why one of the most important steps in learning how to write shell scripts is to learn how to ensure that your scripts are secure—even if it's just your own home computer and an old PC you've converted into a Linux-based media server with Plex or similar.

Let's have a look at some of the basics.

Know the Utilities You Invoke

Here's a classic trojan horse attack: an attacker drops a script called ls into /tmp, and it simply checks to see the userid that invoked it, then hands off its entire argument sequence to the real /bin/ls. If it recognizes userid = root, it makes a copy of /bin/sh into /tmp with an innocuous name, then changes its permission to setuid root.

This is super easy to write. Here's a version off the top of my head:


#!/bin/sh

if [ "$USER" = "root" ] ; then
  /bin/cp /bin/sh /tmp/.secretshell
  /bin/chown root /tmp/.secretshell
  /bin/chmod 4666 root /tmp/.secretshell
fi

exec /bin/ls $*

I hope you understand what just happened. This simple little script has created a shell that always grants its user root access to the Linux system. Yikes. Fancier versions would remove themselves once the root shell has been created, leaving no trace of how this transpired.

Go to Full Article

Simulate Typing with This C Program

Jim Hall — Wed, 24 Oct 2018 12:00:00 +0000

by Jim Hall

I recently created a video demonstration of how to do some work at the command line, but as I tried to record my video, I kept running into problems. I'm just not the kind of person who can type commands at a keyboard and talk about it at the same time. I quickly realized I needed a way to simulate typing, so I could create a "canned" demonstration that I could narrate in my video.

After doing some searching, I couldn't find a command on my distribution that would simulate typing. I wasn't surprised; that's not a common thing people need to do. So instead, I rolled my own program to do it.

Writing a program to simulate typing isn't as difficult as it first might seem. I needed my program to act like the echo command, where it displayed output given as command-line parameters. I added command-line options so I could set a delay between the program "typing" each letter, with an additional delay for spaces and newlines. The program basically did this the following for each character in a given string:

Insert a delay.
Print the character.
Flush the output buffer so it shows up on screen.

First, I needed a way to simulate a delay in typing, such as someone typing slowly, or pausing before typing the next word or pressing Enter. The C function to create a delay is usleep(useconds_t usec). You use usleep() with the number of microseconds you want your program to pause. So if you want to wait one second, you would use usleep(1000000).

Working in microseconds means too many zeroes for me to type, so I wrote a simple wrapper called msleep(int millisec) that does the same thing in milliseconds:


int
msleep (int millisec)
{
  useconds_t usec;
  int ret;


  /* wrapper to usleep() but values in milliseconds instead */


  usec = (useconds_t) millisec *1000;
  ret = usleep (usec);
  return (ret);
}

Next, I needed to push characters to the screen after each delay. Normally, you can use putchar(int char) to send a single character to standard output (such as the screen), but you won't actually see the output until you send a newline. To get around this, you need to flush the output buffer manually. The C function fflush(FILE *stream) will flush an output stream for you. If you put a delay() before each fflush(), it will appear that someone is pausing slightly between typing each character.

Go to Full Article

Code Review--an Excerpt from VM Brasseur's New Book Forge Your Future with Open Source

VM Brasseur — Thu, 18 Oct 2018 12:00:00 +0000

by VM Brasseur

Even new programmers can provide a lot of value with their code reviews. You don't have to be a Rockstar Ninja 10x Unicorn Diva programmer with years and years of experience to have valuable insights. In fact, you don't even have to be a programmer at all. You just have to be knowledgable enough to spot patterns. While you won't be able to do a complete review without programming knowledge, you may still spot things that could use some work or clarification.

If you're not a Rockstar Ninja 10x Unicorn Diva programmer, not only is your code review feedback still valuable, but you can also learn a great deal in the process: Code layout, programming style, domain knowledge, best practices, neat little programming tricks you'd not have seen otherwise, and sometimes antipatterns (or "how not to do things"). So don't let the fact that you're unfamiliar with the code, the project, or the language hold you back from reviewing code contributions. Give it a go and see what there is to learn and discover.

"But," you may wail, "how is that even possible?! I don't know how to program very well! How could I ever do anything valuable on a code review?" Calm yourself, friend. You have a lot to offer here. Earlier I mentioned pattern-spotting, and that's a good place to start. If the contribution you're reviewing looks a lot more complicated than everything around it, you've just spotted a potential problem. Does the code use different indentations or variable naming than elsewhere in the file? That's another potential problem. Is the code contribution really long, when everything else in the file is much shorter? That could be a sign something is wrong. You don't have to be that Rockstar Ninja 10x Unicorn Diva programmer to spot these things; you only have to be familiar with programming and—most importantly—you only have to be looking at the code.

Do be careful as you start code review for a project with which you're not very familiar. Some projects would rather not receive reviews from people who aren't yet skilled in the code in question, as those reviews often can contain errors or inconsistencies with how the project typically operates. Inexperienced reviewers also can confuse inexperienced contributors, who might not know that the person providing feedback to them is not very familiar with the code or the project. Always check the CONTRIBUTING file or ask a core contributor before you start reviewing code contributions, rather than risk stepping on toes or providing feedback when none is wanted.

Go to Full Article

Creating the Concentration Game PAIRS with Bash, Part II

Dave Taylor — Wed, 10 Oct 2018 12:00:00 +0000

by Dave Taylor

Dave finishes up the PAIRS concentration game, only to realize it's too hard to solve!

In my last article, I tossed away my PC card and talked about how I was a fan of the British colonial-era writer Rudyard Kipling. With that in mind, I do appreciate that you're still reading my column.

I was discussing the memory game that the British spy plays with the orphan boy Kim in the book of the same name. The game in question involves Kim being shown a tray of stones of various shapes, sizes and colors. Then it's hidden, and he has to recite as many patterns as he can recall.

The card game Concentration is clearly inspired by the same pattern memorization game, and it's considerably easier to set up: shuffle a deck of cards, place them face down in a grid, then flip pairs to find matches. In the beginning, it's just guessing, of course, but as the game proceeds, it becomes more about spatial memory than luck. Someone with an eidetic memory always will win.

Using letters makes things easy, so I suggested a row, column, notational convention like this:


    1   2   3   4   5   6   7   8   9   10  11  12  13
1: [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-]
2: [-] [-] [-] [A] [-] [-] [-] [-] [-] [-] [-] [-] [-]
3: [-] [-] [-] [-] [-] [-] [-] [-] [E] [-] [-] [-] [-]
4: [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [Z]

You can represent uppercase letters as a shell array like this:


declare -a letters=(A B C D E F G H I J K L M N O P Q R
                    S T U V W X Y Z)

Unfortunately, Bash doesn't support multidimensional arrays, so you're going to have to represent the grid as a one-dimensional array. It's not too hard though, because the grid is straightforward. Here's an index formula if firstvalue is the first digit and rest is the remainder of the index value:


index=$(( ( ( $firstvalue - 1 ) * 13 ) + $rest ))

The letter "E" in the above grid, at 3,9, would show up in the array as ((3-1)*13)+9 or slot 35.

Shuffle Those Values

The script from my last article already initializes everything in sequential order and defaults to 2 * 13 slots (for simplicity in debugging). The work of the script is really in the shuffle, but it turns out that there's a pretty elegant little shuffle algorithm (shown in a kind of sloppy C for illustrative purposes) floating around the internet that can be tapped for this task:

Go to Full Article

Introducing Genius, the Advanced Scientific Calculator for Linux

Joey Bernard — Fri, 05 Oct 2018 11:30:00 +0000

by Joey Bernard

Genius is a calculator program that has both a command-line version and a GNOME GUI version. It should available in your distribution's package management system. For Debian-based distributions, the GUI version and the command-line version are two separate packages. Assuming that you want to install both, you can do so with the following command:


sudo apt-get install genius gnome-genius

If you use Ubuntu, be aware that the package gnome-genius doesn't appear to be in Bionic. It's in earlier versions (trusty, xenial and arty), and it appears to be in the next version (cosmic). I ran into this problem, and thought I'd mention it to save you some aggravation.

Starting the command-line version provides an interpreter that should be familiar to Python or R users.

Figure 1. When you start Genius, you get the version and some license information, and then you'll see the interpreter prompt.

If you start gnome-genius, you'll see a graphical interface that is likely to be more comfortable to new users. For the rest of this article, I'm using the GUI version in order to demonstrate some of the things you can do with Genius.

Figure 2. The GUI interface provides easy menu access to most of the functionality within Genius.

You can use Genius just as a general-purpose calculator, so you can do things like:


genius> 4+5
= 9

Along with basic math operators, you also can use trigonometric functions. This command gives the sine of 45 degrees:


genius> sin(45)
= 0.850903524534

These types of calculations can be of essentially arbitrary size. You also can use complex numbers out of the box. Many other standard mathematical functions are available as well, including items like logarithms, statistics, combinatorics and even calculus functions.

Along with functions, Genius also provides control structures like conditionals and looping structures. For example, the following code gives you a basic for loop that prints out the sine of the first 90 degrees:


for i = 1 to 90 do (
   x = sin(i);
   print(x)
)

As you can see, the syntax is almost C-like. At first blush, it looks like the semicolon is being used as a line-ending character, but it's actually a command separator. That's why there is a semicolon on the line with the sine function, but there is no semicolon on the line with the print function. This means you could write the for loop as the following:

Go to Full Article

Understanding Bash: Elements of Programming

Vladimir Likic — Fri, 28 Sep 2018 14:22:26 +0000

by Vladimir Likic

Ever wondered why programming in Bash is so difficult? Bash employs the same constructs as traditional programming languages; however, under the hood, the logic is rather different.

The Bourne-Again SHell (Bash) was developed by the Free Software Foundation (FSF) under the GNU Project, which gives it a somewhat special reputation within the Open Source community. Today, Bash is the default user shell on most Linux installations. Although Bash is just one of several well known UNIX shells, its wide distribution with Linux makes it an important tool to know.

The main purpose of a UNIX shell is to allow users to interact effectively with the system through the command line. A common shell action is to invoke an executable, which in turn causes the kernel to create a new running process. Shells have mechanisms to send the output of one program as input into another and facilities to interact with the filesystem. For example, a user can traverse the filesystem or direct the output of a program to a file.

Although Bash is primarily a command interpreter, it's also a programming language. Bash supports variables, functions and has control flow constructs, such as conditional statements and loops. However, all of this comes with some unusual quirks. This is because Bash attempts to fulfill two roles at the same time: to be a command interpreter and a programming language—and there is tension between the two.

All UNIX shells, including Bash, are primarily command interpreters. This trait has a deep history, stretching all the way to the very first shell and the first UNIX system. Over time, UNIX shells acquired the programming capabilities by evolution, and this has led to some unusual solutions for the programming environment. As many people come to Bash already having some background in traditional programming languages, the unusual perspective that Bash takes with programming constructs is a source of much confusion, as evidenced by many questions posted on Bash forums.

In this article, I discuss how programming constructs in Bash differ from traditional programming languages. For a true understanding of Bash, it's useful to understand how UNIX shells evolved, so I first review the relevant history, and then introduce several Bash features. The majority of this article shows how the unusual aspects of Bash programming originate from the need to blend the command interpreter function seamlessly with the capabilities of a programming language.

Go to Full Article