Tech Tips

Command-Line Tip: Put Down the Pipe

Kyle Rankin — Tue, 22 Jan 2019 12:30:00 +0000

Learn a few techniques for avoiding the pipe and making your command-line commands more efficient.

Anyone who uses the command line would acknowledge how powerful the pipe is. Because of the pipe, you can take the output from one command and feed it to another command as input. What's more, you can chain one command after another until you have exactly the output you want.

Pipes are powerful, but people also tend to overuse them. Although it's not necessarily wrong to do so, and it may not even be less efficient, it does make your commands more complicated. More important though, it also wastes keystrokes! Here I highlight a few examples where pipes are commonly used but aren't necessary.

Stop Putting Your Cat in Your Pipe

One of the most common overuses of the pipe is in conjunction with cat. The cat command concatenates multiple files from input into a single output, but it has become the overworked workhorse for piped commands. You often will find people using cat just to output the contents of a single file so they can feed it into a pipe. Here's the most common example:


cat file | grep "foo"

Far too often, if people want to find out whether a file contains a particular pattern, they'll cat the file piped into a grep command. This works, but grep can take a filename as an argument directly, so you can replace the above command with:


grep "foo" file

The next most common overuse of cat is when you want to sort the output from one or more files:


cat file1 file2 | sort | uniq

Like with grep, sort supports multiple files as arguments, so you can replace the above with:


sort file1 file2 | uniq

In general, every time you find yourself catting a file into a pipe, re-examine the piped command and see whether it can accept files directly as input first either as direct arguments or as STDIN redirection. For instance, both sort and grep can accept files as arguments as you saw earlier, but if they couldn't, you could achieve the same thing with redirection:


sort < file1 file2 | uniq
grep "foo" < file

Remove Files without xargs

The xargs command is very powerful on the command line—in particular, when piped to from the find command. Often you'll use the find command to pick out files that have a certain criteria. Once you have identified those files, you naturally want to pipe that output to some command to operate on them. What you'll eventually discover is that commands often have upper limits on the number of arguments they can accept.

Go to Full Article

Top 12 Tech Tips from 2018

Carlie Fairchild — Mon, 24 Dec 2018 12:01:41 +0000

by Carlie Fairchild

Go to Full Article

Travel Laptop Tips in Practice

Kyle Rankin — Wed, 21 Nov 2018 13:00:00 +0000

by Kyle Rankin

It's one thing to give travel advice; it's another to follow it.

In past articles, I've written about how to prepare for a vacation or other travel when you're on call. And, I just got back from a vacation where I put some of those ideas into practice, so I thought I'd write a follow-up and give some specifics on what I recommended, what I actually did and how it all worked.

Planning for the Vacation

The first thing to point out is that this was one of the first vacations in a long time where I was not on call, directly or indirectly. In my long career as a sysadmin responsible for production infrastructure, I've almost always been on call (usually indirectly) when on vacation. Even if someone else was officially taking over on-call duties while I was away, there always was the risk that a problem would crop up where they would need to escalate up to me. Often on my vacations something did blow up to the point that I needed to get involved. I've now transitioned into more of a management position, so the kinds of emergencies I face are much different.

I bring up the fact that I wasn't on an on-call rotation not because it factored into how I prepared for the trip, but because, generally speaking, it didn't factor in except that I didn't have to go to as extreme lengths to make sure everyone knew how to contact me in an emergency. Even though I wasn't on call, there still was a chance, however remote, that some emergency could pop up where I needed to help. And, an emergency might require that I access company resources, which meant I needed to have company credentials with me at a minimum. I imagine for most people in senior-enough positions that this would also be true. I could have handled this in a few ways:

Hope that I could access all the work resources I might need from my phone.
Carry a copy of my password manager database with me.
Put a few select work VMs on my travel laptop.

I chose option number 3, just to be safe. Although I'm not superstitious, I still figured that if I were prepared for an emergency, there was a better chance one wouldn't show up (and I was right). At the very least, if I were well prepared for a work emergency, if even a minor problem arose, I could respond to it without a major inconvenience instead of scrambling to build some kind of MacGyver-style work environment out of duct tape and hotel computers.

Selecting the Travel Computer

As I've mentioned in previous articles, I recommend buying a cheap, used computer for travel. That way, if you lose it or it gets damaged, confiscated or stolen, you're not out much money. I personally bought a used Acer Parrot C710 for use as a travel computer, because it's small, cheap and runs QubesOS pretty well once you give it enough RAM.

Go to Full Article

Simulate Typing with This C Program

Jim Hall — Wed, 24 Oct 2018 12:00:00 +0000

by Jim Hall

I recently created a video demonstration of how to do some work at the command line, but as I tried to record my video, I kept running into problems. I'm just not the kind of person who can type commands at a keyboard and talk about it at the same time. I quickly realized I needed a way to simulate typing, so I could create a "canned" demonstration that I could narrate in my video.

After doing some searching, I couldn't find a command on my distribution that would simulate typing. I wasn't surprised; that's not a common thing people need to do. So instead, I rolled my own program to do it.

Writing a program to simulate typing isn't as difficult as it first might seem. I needed my program to act like the echo command, where it displayed output given as command-line parameters. I added command-line options so I could set a delay between the program "typing" each letter, with an additional delay for spaces and newlines. The program basically did this the following for each character in a given string:

Insert a delay.
Print the character.
Flush the output buffer so it shows up on screen.

First, I needed a way to simulate a delay in typing, such as someone typing slowly, or pausing before typing the next word or pressing Enter. The C function to create a delay is usleep(useconds_t usec). You use usleep() with the number of microseconds you want your program to pause. So if you want to wait one second, you would use usleep(1000000).

Working in microseconds means too many zeroes for me to type, so I wrote a simple wrapper called msleep(int millisec) that does the same thing in milliseconds:


int
msleep (int millisec)
{
  useconds_t usec;
  int ret;


  /* wrapper to usleep() but values in milliseconds instead */


  usec = (useconds_t) millisec *1000;
  ret = usleep (usec);
  return (ret);
}

Next, I needed to push characters to the screen after each delay. Normally, you can use putchar(int char) to send a single character to standard output (such as the screen), but you won't actually see the output until you send a newline. To get around this, you need to flush the output buffer manually. The C function fflush(FILE *stream) will flush an output stream for you. If you put a delay() before each fflush(), it will appear that someone is pausing slightly between typing each character.

Go to Full Article

What's Your System's Uptime?

Ricardo Fraile — Wed, 03 Oct 2018 12:00:00 +0000

by Ricardo Fraile

Keep track of your system's uptime and downtime with the tuptime tool.

Finding your system's uptime is easy if the "beginning" means the last startup; the historical uptime command reports that information. But what happens if by "beginning" you mean the first startup ever of the system? Or the last 365 days? Or the last month?

Is there any way to have an accumulated uptime—or even better, a look at the whole system's life? For example, cars have odometers, and you can see the miles/kilometers since the first day. For computers, a tool was developed exactly for this task: tuptime.

tuptime reports the historical and statistical running and stopped time of your system, keeping track between restarts. Its main goals are:

Count system startups.
Register the first boot time (since installation).
Count intended and accidental shutdowns.
Show the uptime and downtime percentage since the first boot time.
Show the accumulated system uptime, downtime and total.
Show the longest, shortest and average uptime and downtime.
Show the current uptime.
Print a formatted table or list with most of the previous values.
Register used kernels.
Create reports since and/or until a given startup or timestamp.
Create reports in CSV format.

It works very simply. tuptime falls to the init manager for execution at startup and shutdown, and then into a cron task that launches regular executions in the meantime—there isn't any dæmon to worry about. Internally, it looks at the btime value (available in /proc/stat) and the uptime value (from /proc/uptime), and that's basically it.

The installation process is easy in Debian, Ubuntu and derivative distributions, using their respective package managers, and it should be available in all the official repositories. As prerequisites, it needs Python 3 and the SQLite library, which usually are included in core packages by default.

Once it's available on your system, you can get the information. It has three output formats: the default is a summary, and there also are table and list outputs to print the registered behavior.

Figure 1. Example tuptime Execution after Installation

The first execution reports the time since the system was booted, and the lines are self-explanatory (note that the date format is based on the system's locale settings):

Go to Full Article

Writing More Compact Bash Code

Mitch Frazier — Tue, 18 Sep 2018 12:00:00 +0000

by Mitch Frazier

In any programming language, idioms may be used that may not seem obvious from reading the manual. Often these usages of the language represent ways to make your code more compact (as in requiring fewer lines of code). Of course, some will eschew these idioms believing they represent bad style. Style, of course, is in the eyes of beholder, and this article is not intended as an exercise in defining good or bad style. So for those who may be tempted to comment on the grounds of style I would (re)direct your attention to /dev/null.

Go to Full Article

Edit PDFs with Xournal

Kyle Rankin — Thu, 06 Sep 2018 12:00:00 +0000

by Kyle Rankin

Forget all of those magical command-line PDF incantations and edit your PDFs easily with Xournal.

Somehow, despite all the issues with proprietary clients and the history of security issues with Acrobat, PDFs have become the de facto standard for your average print-ready document shared around the office. Sure, people might use some kind of open document format or a cloud editor if they intend to edit a document, but if the goal is to print the document or lock its contents in place, most people these days will export it to a PDF.

Reading PDFs is typically fine on Linux, because Linux has plenty of applications that can open PDFs for viewing, and you easily can print PDFs under Linux as well. Even Adobe supplied a proprietary (and somewhat outdated) port of its Acrobat Reader for Linux. Some distributions also offer the ability to create a special software printer that converts any print job sent to it into a local PDF file.

The problem comes when people want to turn read-only print-ready PDFs into read-write documents you need to modify. As more people work in paperless offices with strictly digital documents and fewer people own fax machines, you are more likely to find official documents like contracts show up in your INBOX in PDF format. These contracts likely were created with a proprietary PDF editor tool, and they usually have blanks for you to fill in and often signature lines so you can add a real signature. Unfortunately, for the longest time, even if you were using Adobe's own Linux port of Acrobat Reader, you couldn't reliably edit these PDFs, and you certainly couldn't easily add a real signature.

A lot of Linux applications claim the ability to edit PDFs from graphical tools like GIMP, or the aforementioned Acrobat Reader or tools like Inkscape. In the past, I've even gone so far as to use command-line tools to convert a PDF into multiple pages of a different format, edit that format, then use the command-line tools to convert it back to a PDF.

Then I discovered Xournal. Xournal is a graphical tool that's designed for note-taking and sketching either with a keyboard and mouse or even with a tablet and stylus. This program is pretty common, and you should be able to install it in any major Linux distribution, but otherwise, you can download the software from its Sourceforge page.

Go to Full Article

Copy and Paste in Screen

Kyle Rankin — Thu, 23 Aug 2018 13:08:08 +0000

by Kyle Rankin

Put the mouse down, and copy and paste inside a terminal with your keyboard using Screen.

Screen is a command-line tool that lets you set up multiple terminal windows within it, detach them and reattach them later, all without any graphical interface. This program has existed since before I started using Linux, but first I clearly need to address the fact that I'm even using Screen at all prior to writing a tech tip about it. I can already hear you ask, "Why not tmux?" Well, because every time someone tries to convince me to make the switch, it's usually for one of the following reasons:

Screen isn't getting updates: I've been happy with the current Screen feature set for more than a decade, so as long as distributions continue to package it, I don't feel like I need any newer version.
tmux key bindings are so much simpler: I climbed the Screen learning curve more than a decade ago, so to me, the Screen key bindings are second nature.
But you can do vertical and horizontal splits in tmux: you can do them in Screen too, and since I climbed that learning curve ages ago, navigating splits are part of my muscle memory just like inside vim.

So now that those arguments are out of the way, I thought those of you still using Screen might find it useful to learn how to do copy and paste within Screen itself. Although it's true that you typically can use your mouse to highlight text and paste it, if you are a fan of staying on the home row like I am, you realize how much faster and more efficient it is if you can copy and paste from within Screen itself using the keyboard. In fact, I found that once I learned this method, I ended up using it multiple times every day.

Enter Copy Mode

The first step is to enter copy mode from within Screen. Press Ctrl-a-[ to enter copy mode. Once you're in this mode, you can use arrow keys or vi-style keybindings to navigate up and down your terminal window. This is handy if you are viewing a log or other data that has scrolled off the screen and you want to see it. Typically people who are familiar with copy mode just use it for scrolling and then press q to exit that mode, but once you are in copy mode, you also can move the cursor to an area you want to copy.

Copy Text

To copy text once in copy mode, move your cursor to where you want to start to copy and then press the space bar. This will start the text selection, and you'll see the cursor change so that it highlights the text as you then move the cursor to select everything you want to copy. Once you are done selecting text, press the space bar again, and it will be copied to Screen's copy buffer. Once text is copied to Screen's clipboard, it automatically will exit copy mode.

Go to Full Article

Telecommuting Tips

Kyle Rankin — Fri, 10 Aug 2018 13:30:00 +0000

by Kyle Rankin

With all the collaboration technology available for offices today, there's no reason telecommuters can't be as productive and as connected as other team members.

I live in the San Francisco Bay Area, known for high-tech companies, horrible traffic and high cost of living. When it came time for me to buy a house, I chose an area that left me with a 90–120-minute commute, depending on traffic and the time of day, so through the years, I've negotiated work-from-home days and have experience with telecommuting at companies of various sizes with different proportions of remote workers. Telecommuting is not only more convenient for many employees, it also can get the best work out of people, because it can grant better opportunities to focus and lets employees get right to work instead of spending hours getting to and from work. Unfortunately, many places inadvertently sabotage their telecommuters with bad practices, so here are a few tips to help make telecommuting successful.

Invest in Good Teleconference Hardware

I've attended many video conferences where the audio was so horrible, I might as well have not joined. Or worse, there was a time when one speaker was loud and clear, but when the conversation went to the other side of the table, it was inaudible. Although it's nice to have quality cameras, having quality microphones is critical. Make sure each of your meeting rooms has quality microphones that can pick up sounds all around the meeting table, and make sure attendees speak up. Relying on the microphone on someone's laptop just doesn't cut it for meetings involving more than two people. Although it's considered good meeting etiquette to have only one person speak at a time, this protocol is extra important if you have anyone calling in, as cross-talk makes it all but impossible to hear either conversation even over a good microphone.

Add Video Conference Links to Every Meeting

Make it a habit to add a link to your video conference room for each meeting you create, even if all of the attendees are expected to be in the office. This habit ensures that when you realize you forgot to invite a remote workers, you aren't scrambling to figure out how to set up the video conference, plus sometimes even team members in the office need to work from home at the last minute. If your scheduling software can do this automatically, even better (some do this by having each meeting room in a contact list and inviting the relevant meeting room to the meeting). Also make sure you set this up for all-hands company-wide meetings.

Go to Full Article

Clearing Out /boot

Adam McPartlan — Fri, 29 Jun 2018 12:00:00 +0000

by Adam McPartlan

The /boot partition sometimes needs a bit of attention. If you enable automatic updates, it will fill up with old kernels that you'll probably never need. It also will stop you from running aptitude to install or remove anything. If you find yourself in this situation, you can use dpkg to get around it. dpkg is the higher-level package manager in Debian-based distributions, and it's very useful when aptitude has broken.

To see the status of your partitions, run: df -h:


Filesystem      Size  Used Avail Use% Mounted on
udev            3.0G   12K  3.0G   1% /dev
tmpfs           597M  528K  597M   1% /run
/dev/dm-0        97G   14G   78G  15% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            3.0G     0  3.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sda1       228M  219M     0 100% /boot

If you look in the directory /boot, you will see it full of old kernels and images. It is not advisable just to delete them, as you can break your system. Run uname -r, which will tell you what kernel you are currently on:


3.13.0-137-generic

Let's find out which kernels are installed and which can be purged from your system. To do this, run the following:


dpkg --list "linux-image*" | grep -v $(uname) | grep ii

This will use dpkg to list all Linux kernel images (excluding the one you are using) that are installed.

The output still might be quite long, so let's refine it by piping the results in to awk. The awk command below is an instruction to print the second column from the output:


dpkg --list "linux-image*" | grep -v $(uname -r) |
 ↪grep ii | awk '{ print $2 }'

This provides a list to work with, and you can stick it in a script or run it from the command line to purge them all.

Caution: make sure the kernel you are using is not in the list. You should have eliminated that when you specified grep -v $(uname -r). The -v tells grep to exclude anything that contains the output of uname -r.

If you are happy and have sudo privileges, go ahead:


sudo dpkg --purge $(dpkg --list "linux-image*" | grep -v
 ↪$(uname -r) | grep ii | awk '{ print $2 }')

To finish off, run sudo update-grub2. This will ensure that grub is updated with the available kernels. Otherwise, you may be heading for trouble. Then fix aptitude by running: sudo apt-get -f install, followed by sudo apt-get auto remove to clear the images out of aptitude.

Look at your partition, and you will see that it has free space:

Go to Full Article