GitHub

Writing GitHub Web Hooks with Bash

Andy Carlson — Wed, 07 Aug 2019 16:30:00 +0000

Bring your GitHub repository to the next level of functionality.

For the past year since Microsoft has acquired GitHub, I've been hosting my Git repositories on a private server. Although I relished the opportunity and challenge of setting it all up, and the end product works well for my needs, doing this was not without its sacrifices. GitHub offers a clean interface for configuring many Git features that otherwise would require more time and effort than simply clicking a button. One of the features made easier to implement by GitHub that I was most fond of was web hooks. A web hook is executed when a specific event occurs within the GitHub application. Upon execution, data is sent via an HTTP POST to a specified URL.

This article walks through how to set up a custom web hook, including configuring a web server, processing the POST data from GitHub and creating a few basic web hooks using Bash.

Preparing Apache

For the purpose of this project, let's use the Apache web server to host the web hook scripts. The module that Apache uses to run server-side shell scripts is mod_cgi, which is available on major Linux distributions.

Once the module is enabled, it's time to configure the directory permissions and virtual host within Apache. Use the /opt/hooks directory to host the web hooks, and give ownership of this directory to the user that runs Apache. To determine the user running an Apache instance, run the following command (provided Apache is currently running):


ps -e -o %U%c| grep 'apache2\|httpd'

These commands will return a two-column output containing the name of the user running Apache and the name of the Apache binary (typically either httpd or apache2). Grant directory permission with the following chown command (where USER is the name of the user shown in the previous ps command):


chown -R USER /opt/hooks

Within this directory, two sub-directories will be created: html and cgi-bin. The html folder will be used as a web root for the virtual host, and cgi-bin will contain all shell scripts for the virtual host. Be aware that as new sub-directories and files are created under /opt/hooks, you may need to rerun the above chown to verify proper access to files and sub-directories.

Here's the configuration for the virtual host within Apache:

Go to Full Article

Spy Games: the NSA and GCHQ Offer Their Software to the Open Source Community

David Habusha — Thu, 07 Mar 2019 12:30:00 +0000

by David Habusha

Spies worth their salt are generally expected to be good at keeping secrets. With dead drops, encryption, cyanide pills and the like, openly sharing useful information isn’t supposed to be a part of the job description.

So it caught more than a few of us off guard when a couple years ago, some of the top spy agencies began contributing code to GitHub, making it available to the masses by open-sourcing some of their software.

The National Security Agency, the American signals intelligence organization that is tasked with the majority of the cyber-snooping, has released two separate pages on GitHub. The first is the NSA's primary account on GitHub that has 17 listed repos, followed up by its more substantive “NSA Cybersecurity” page with its 31 repositories.

Even though the NSA appears to have been posting some of its software as open source since 2017, presumably a result in part of the effort from the US government to make more of the code produced by the USG available to the public, the agency made news in early January when it announced plans to release a new product to the Open Source community.

The software is called GHIDRA, and it has been described as a tool for reverse-engineering malware. According to reports, GHIDRA has been referenced in the past during the Vault7 document leaks and is available for use across all the major operating systems. Those who are curious for more information on this tool and how to use it can catch a glimpse at a demonstration that the NSA has committed to putting on at this year’s RSA conference.

However, with perhaps less fanfare, it would seem as though it was the Brits who first made the move to take some of their code open source. The British SigInt agency GCHQ released its first piece of open-source tooling with the Gaffer graph database back in 2015, beating the Americans by two years. At the time of writing, the good folks at Her Majesty’s cyber-snooping agency have 39 repositories on offer for all to try out, including one called the CyberChef, which is billed as the “Cyber Swiss Army Knife—a web app for encryption, encoding, compression, and data analysis”.

Go to Full Article

Take Your Git In-House

John S. Tonello — Thu, 27 Sep 2018 12:56:39 +0000

by John S. Tonello

If you're wary of the Microsoft takeover of GitHub, or if you've been looking for a way to ween yourself off free public repositories, or if you want to ramp up your DevOps efforts, now's a good time to look at installing and running GitLab yourself. It's not as difficult as you might think, and the free, open-source GitLab CE version provides a lot of flexibility to start from scratch, migrate or graduate to more full-fledged versions.

In today's software business, getting solid code out the door fast is a must, and practices to make that easier are part of any organization's DevOps toolset. Git has risen to the top of the heap of version control tools, because it's simple, fast and makes collaboration easy.

For developers, tools like Git ensure that their code isn't just backed up and made available to others, but nearly guarantees that it can be incorporated into a wide variety of third-party development tools—from Jenkins to Visual Studio—that make continuous integration and continuous delivery (CI/CD) possible. Orchestration, automation and deployment tools easily integrate with Git as well, which means code developed on any laptop or workstation anywhere can be merged, branched and integrated into deployed software. That's why version control repositories are the future of software development and DevOps, no matter how big or small you are, and no matter whether you're building monolithic apps or containerized ones.

Getting Started with Git

Git works by taking snapshots of code on every commit, so every version of contributed code is always available. That means it's easy to roll back changes or look over different contributors' work.

If you're working in an environment that uses Git, you can do your work even when you're offline. Everything is saved in a project structure on your workstation, just as it is in the remote Git repository, and when you're next online, your commits and pushes update the master (or other) code branch quickly and easily.

Most Git users (even newbies) use the Git command-line tools to clone, commit and push changes, because it's easy, and for nearly 28-million developers, GitHub has become the de facto remote Git-based repository for their work. In fact, GitHub has moved beyond being just a code repository to become a multifaceted code community featuring 85-million projects. That's a lot of code.

Go to Full Article

Building a Bare-Bones Git Environment

Andy Carlson — Tue, 24 Jul 2018 12:00:00 +0000

by Andy Carlson

How to migrate repositories from GitHub, configure the software and get started with hosting Git repositories on your own Linux server.

With the recent news of Microsoft's acquisition of GitHub, many people have chosen to research other code-hosting options. Self-hosted solutions like GitLabs offer a polished UI, similar in functionality to GitHub but one that requires reasonably well-powered hardware and provides many features that casual Git users won't necessarily find useful.

For those who want a simpler solution, it's possible to host Git repositories locally on a Linux server using a few basic pieces of software that require minimal system resources and provide basic Git functionality including web accessibility and HTTP/SSH cloning.

In this article, I show how to migrate repositories from GitHub, configure the necessary software and perform some basic operations.

Migrating Repositories

The first step in migrating away from GitHub is to relocate your repositories to the server where they'll be hosted. Since Git is a distributed version control system, a cloned copy of a repository contains all information necessary for running the entire repository. As such, the repositories can be cloned from GitHub to your server, and all the repository data, including commit logs, will be retained. If you have a large number of repositories this could be a time-consuming process. To ease this process, here's a bash function to return the URLs for all repositories hosted by a specific GitHub user:


genrepos() {
    if [ -z "$1" ]; then
        echo "usage: genrepos "
    else
        repourl="https://github.com/$1?tab=repositories"
        while [ -n "$repourl" ]; do
            curl -s "$repourl" | awk '/href.*codeRepository/
 ↪{print gensub(/^.*href="\/(.*)\/(.*)".*$/,
↪"https://github.com/\\1/\\2.git","g",$0); }'
            export repourl=$(curl -s "$repourl" | grep'>Previous<.
↪*href.*>Next<' | grep -v 'disabled">Next' | sed
↪'s/^.*href="//g;s/".*$//g;s/^/https:\/\/github.com/g')
        done
    fi
}

This function accepts a single argument of the GitHub user name. If the output of this command is piped into a while loop to read each line, each line can be fed into a git clone statement. The repositories will be cloned into the /opt/repos directory:

Go to Full Article

Opinion: GitHub vs GitLab

Matt Lee — Mon, 16 Jul 2018 11:30:00 +0000

by Matt Lee

Free software deserves free tools, not Microsoft-owned GitHub.

So, Microsoft bought GitHub, and many people are confused or worried. It's not a new phenomenon when any large company buys any smaller company, and people are right to be worried, although I argue that their timing is wrong. Like Microsoft, GitHub has made some useful contributions to free and open-source software, but let's not forget that GitHub's main product is proprietary software. And, it's not just some innocuous web service either; GitHub makes and sells a proprietary software package you can download and run on your own server called GitHub Enterprise (GHE).

Let's remember how we got here. BitMover made a tool called BitKeeper, a proprietary version control system that allowed free-of-charge licenses to free software projects. In 2002, the Linux kernel switched to using BitKeeper for its version control, although some notable developers made the noble choice to refuse to use the proprietary program. Many others did not, and for a number of years, kernel development was hampered by BitKeeper's restrictive noncommercial licenses.

In 2005, Andrew Tridgell, working at OSDL, developed a client that bypassed this restriction, and as a result, BitMover removed licenses to BitKeeper from all OSDL employees—including Linus Torvalds. Eventually, all non-commercial licenses were stopped, and new licenses included clauses preventing the development of alternative version control systems. As a result of this, two new projects were born: Mercurial and Git. Created in a few short weeks in 2005, Git quickly became the version control system for Linux development.

Proprietary version control tools aren't common in free software development, but proprietary collaboration websites have been around for some time. One of the earliest collaboration websites still around today is Sourceforge. Sourceforge was created in the late 1990s by VA Software, and the code behind the project was released in 2000.

Quickly this situation changed, and the project was shuttered and then became Sourceforge Enterprise Edition, a proprietary software package. The code that ran Sourceforge was forked into GNU Savannah (later Savane) and GForge, and it's still use today by both the GNU Project and CERN. When I last wrote about this problem, almost exactly ten years ago, Canonical's ambitious Launchpad service still was proprietary, something later remedied in 2009. Gitorious was created in 2010 and was for a number of years the Git hosting platform for the discerning free software developer, as the code for Gitorious was fully public and licensed under favorable terms for the new wave of AGPL-licensed projects that followed the FSF's Franklin Street Statement. Gitorious, also, is sadly no longer with us.

Go to Full Article

Git Your July 2018 Issue of Linux Journal: Now Available

Carlie Fairchild — Mon, 02 Jul 2018 17:30:00 +0000

by Carlie Fairchild

Along with Microsoft buying Github recently, we received hundreds of questions and comments about all things git. How does one install and run GitLab themselves? Should they? What's the difference between GitHub and GitLab? How can one migrate repositories from GitHub and host on their own Linux server? So with this July issue of Linux Journal, we take a Deep Dive in to... git. Enjoy!

Feature articles include:

A Git Origin Story by Zack Brown

A look at the Linux kernel developers' various revision control solutions through the years, Linus Torvalds' decision to use BitKeeper and the controversy that followed, and how Git came to be created.

Git Quick Start Guide by Patrick Whelan

Ditch USBs and start using real version control, and if you follow this guide, you can start using git in 30 minutes!

Building a Bare-Bones Git Environment by Andy Carlson

How to migrate repositories from GitHub, configure the software and get started with hosting Git repositories on your own Linux server.

Take Your Git In-House by John S. Tonello

Terrible Ideas in Git by Corey Quinn

This article was derived from a talk that GitHub Universe faithfully rejects every year. I can't understand why....

Opinion: GitHub vs GitLab by Matt Lee

Free software deserves free tools, not Microsoft-owned GitHub.

Other articles in this issue:

Encrypting NFSv4 with Stunnel TLS by Charles Fisher

NFS clients and servers push file traffic over clear-text connections in the default configuration, which is incompatible with sensitive data. TLS can wrap this traffic, finally bringing protocol security. Before you use your cloud provider's NFS tools, review all of your NFS usage and secure it where necessary.

Advertising 3.0 by Doc Searls

Road to RHCA—Preparation Meets Opportunity by Taz Brown

FOSS Project Spotlight: ONLYOFFICE, an Online Office Suite by Tatiana Kochedykova

At Rest Encryption by Kyle Rankin

Progress with Your Image by Kyle Rankin

FOSS Project Spotlight: Pydio Cells, an Enterprise-Focused File-Sharing Solution by Italo Vignoli

Atomic Modeling with GAMGI by Joey Bernard

News Briefs by Jill Franklin

Kyle Rankin's Hack and /: What Really IRCs Me: Slack

Reuven M. Lerner's At the Forge: Introducing Python 3.7's Dataclasses

Go to Full Article

Terrible Ideas in Git

Corey Quinn — Mon, 02 Jul 2018 13:50:25 +0000

by Corey Quinn

This article was derived from a talk that GitHub Universe faithfully rejects every year. I can't understand why....

For better or worse, git has become one of the Open Source community's more ubiquitous tools. It lets you manage code effectively. It helps engineers who are far apart collaborate with each other. At its heart, it's very simple, which is why the diagram in so many blog posts inevitably looks something like the one shown in Figure 1.

Figure 1. Git Model (Source: https://nvie.com)

The unfortunate truth that's rarely discussed in detail is that git has a dark side: it makes us feel dumb. I don't care who you are—we all hit a point wherein we shrug, give up and go scrambling for Stack Overflow (motto: "This thread has been closed as Off Topic") to figure out how best to get out of the terrible situations we've caused for ourselves. The only question is how far down the rabbit hole you can get before the madness overtakes you, and you begin raising goats for a living instead.

At its core, all git does is track changes to files and folders. git commit effectively takes a snapshot of the filesystem (as represented by the items added to the staging area) at a given point in time:


cquinn@1d732dc08938 ~/demo1 % git init
Initialized empty Git repository in /home/cquinn/demo1/.git/
cquinn@1d732dc08938(master|...) ~/demo1 % git add ubuntu.iso
cquinn@1d732dc08938(master|·1) ~/demo1 % git commit
 ↪-m "Initial commit"
[master (root-commit) b0d3bfb] Initial commit
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 ubuntu.iso
cquinn@1d732dc08938(master|✓) ~/demo1 % git rm --cached
 ↪ubuntu.iso
rm 'ubuntu.iso'
cquinn@1d732dc08938(master|·1✓) ~/demo1 % git
 ↪commit -m "There I fixed it"
[master 2d86934] There I fixed it
 1 file changed, 0 insertions(+), 0 deletions(-)
 delete mode 100644 ubuntu.iso
cquinn@1d732dc08938(master|...) ~/demo1 % du -hs .git
174M    .git

So if you do something foolish, such as committing large binaries, you can't just revert the commit—it's still going to live in your git repository. If you've pushed that thing elsewhere, you get to rewrite history forcibly, either with git-filter-branch or the bfg. Either way, it's extra work that's unpleasant to others who share your repository.

Fundamentally, all that git does is create a .git folder in the top level of the repository. This subdirectory contains files and folders that change over time. Wait, isn't there a tool for that?

Go to Full Article

Microsoft Buys GitHub: Three Weeks Later

Marcel Gagné — Tue, 26 Jun 2018 11:30:00 +0000

by Marcel Gagné

I heard that Microsoft would be buying GitHub just a couple days before it happened when Carlie Fairchild at Linux Journal told me about it. I replied to the news with a solid, “Get! Out!” Needless to say, I had my doubts. As someone who remembers all too well the “Embrace, extend and extinguish" days of Microsoft, the news of this latest embrace did, however briefly, bring back those old memories. When I was asked what I thought, I answered that the optics were bad.A lot of years have passed since, back in 2001, Steve Ballmer declared Linux to be a cancer. These days, Microsoft loves Linux. It says so right on its website. Two years ago, Steve Ballmer also proclaimed his love for Linux. In 2018, Microsoft has its own distribution that it uses in its Azure cloud. Microsoft includes several different flavors of Linux in its app store (the Windows Subsystem for Linux), all of which can be installed on Windows 10. Microsoft develops for Linux. Heck, Microsoft even contributes to the Linux kernel.

The reason I felt the optics were bad is that Microsoft has spent the last few years going out of its way to paint itself as a friend to Linux and open source. This, I thought, can only be seen as a bad move. Well, it’s been three weeks, more or less, since the the acquisition became official, to the tune of $7.5 billion US. What happened?

For starters, Jim Zemlin, the Executive Director of the Linux Foundation, praised the Microsoft acquisition of GifHub, suggesting that it could be a good thing for everyone. Former open-source star and the new CEO of GitHub, Nat Friedman, did an AMA on Reddit to reassure developers of open-source software that they had nothing to fear from the new landlords.

Linus Torvalds, the creator of the Linux kernel and git itself, didn’t comment directly, but he has been critical of GitHub in the past, so perhaps it doesn’t change anything for him. Besides, the kernel is primarily housed on a private git server, and GitHub is just a public mirror for the code.

That didn’t stop some number of open-source developers from swearing off GitHub and looking for other places to host their projects. GitLab, a minor competitor to GitHub, seemed poised to be the natural beneficiary of this move. In a Twitter post dated June 3rd, it cited that its GitHub to GitLab conversion rate was running at ten times normal.

Go to Full Article

Back Up GitHub and GitLab Repositories Using Golang

Amit Saha — Mon, 04 Jun 2018 21:46:37 +0000

by Amit Saha

Want to learn Golang and build something useful? Learn how to write a tool to back up your GitHub and GitLab repositories.

GitHub and GitLab are two popular Git repository hosting services that are used to host and manage open-source projects. They also have become an easy way for content creators to be able to invite others to share and collaborate without needing to have their own infrastructure setup.

Using hosted services that you don't manage yourself, however, comes with a downside. Systems fail, services go down and disks crash. Content hosted on remote services can simply vanish. Wouldn't it be nice if you could have an easy way to back up your git repositories periodically into a place you control?

If you follow along with this article, you will write a Golang program to back up git repositories from GitHub and GitLab (including custom GitLab installations). Being familiar with Golang basics will be helpful, but not required. Let's get started!

Hello Golang

The latest stable release of Golang at the time of this writing is 1.8. The package name is usually golang, but if your Linux distro doesn't have this release, you can download the Golang compiler and other tools for Linux. Once downloaded, extract it to /usr/local:


$ sudo tar -C /usr/local -xzf 
$ export PATH=$PATH:/usr/local/go/bin

Opening a new terminal and typing $ go version should show the following:


$ go version
go version go1.8 linux/amd64

Let's write your first program. Listing 1 shows a program that expects a -name flag (or argument) when run and prints a greeting using the specified name. Compile and run the program as follows:


$ go build listing1.go
$ ./listing1 -name "Amit"
Hello Amit

$ ./listing1
./listing1
2017/02/18 22:48:25 Please specify your name using -name
$ echo $?
1

If you don't specify the -name argument, it exits printing a message with a non-zero exit code. You can combine both compiling and running the program using go run:


$ go run listing1.go -name Amit
2017/03/04 23:08:11 Hello Amit

Listing 1. Example Program listing1.go

Go to Full Article