How to customize your terminal prompt (with colors)

At work, the Go projects we're working on have workspaces that get pretty deep and one of the directory names is repeated down the tree. I ended up having no room to type commands in my terminal that didn't wrap to the next line. Annoying!

I could have just limited the prompt to show me the name of the current directory and use pwd when needed, but that was kind of a pain. Then I watched this talk and saw his terminal, and got inspired.

So here's my terminal and prompt now:

Customized terminal prompt -- the red "root" text blinks as a warning/reminder

Doing this took a while to figure out (especially the colors, and getting the root user to share the look). It's a simple idea: edit your ~/.bash_profile file and specify the PS1 variable.

Here's how I did it (this works in both my Mac and Raspberry Pi):

Decoration1="\[\e[90m\]╔["
RegularUserPart="\[\e[36m\]\u"
RootUserPart="\[\e[31;5m\]\u\[\e[m\]"
Between="\[\e[90m\]@"
HostPart="\[\e[32m\]\h:"
PathPart="\[\e[93;1m\]\w"
Decoration2="\[\e[90m\]]\n╚>\[\e[m\]"
case `id -u` in
    0) export PS1="$Decoration1$RootUserPart$Between$HostPart$PathPart$Decoration2# ";;
    *) export PS1="$Decoration1$RegularUserPart$Between$HostPart$PathPart$Decoration2$ ";;
esac

Each segment that looks like \[\e[90m\] changes the color and attributes of the following text, according to an enumeration like this one. The following text keeps those settings until you change it again or reset it. The number before the m is the color. Sometimes there's a number, semicolon, and another number. The second number in that case specifies underline, bold, blink, etc. You can omit both entirely to reset the text to its default or previous style.

To get the root user to share the style, you can, if on a Mac, put that code in your /etc/bashrc file, or on any Linux system (including RPi), you can have the root user's .bashrc source your own .bashrc file. A simple line similar to the following should do the trick (include the dot at the beginning of the line; or you can replace it with source):

. /home/pi/.bashrc

You may also like to have a blank line before each prompt to make it feel a little more "roomy" and less cluttered. You can easily do this by adding \n at the beginning of the $Decoration1 value.

I've really enjoyed this new prompt style. I have plenty of room to type, can clearly see my user, hostname, and current path at a glance. My prompts don't get lost in lots of output (like a disastrous g++ compile).

You're welcome to use it and customize it however you'd like. I'd be interested to see what you come up with.

Read More

Automatically make a Raspberry Pi with wifi support

Okay, I love my Raspberry Pi, but setting it up just the way I want got so involved I was afraid I couldn't do it again if I had to. So I wrote a script to automate it, and decided to publish it.

So stop baking your Pi manually. MakeMyPi automates the process. It even configures the WiFi for you (assuming you're using a dongle that's supported by the OS, Raspbian Wheezy).

Visit the project on GitHub

or

View an asciicast of the script in action

MakeMyPi in action

(Sorry, but right now, it only works from a Mac.)

All you have to do is:

1. Make sure your own public key is in ~/.ssh/id_rsa.pub (which is the default location)
2. Follow the easy configuration instructions
3. Run the script and follow instructions; it will ding at you when it needs your attention

MakeMyPi can do the following for your Raspberry Pi:

• Download an operating system image for you, if necessary
• Write the operating system image file to the SD card
• Install its own public/private key pair that you provide it
• Authorize your own public key to log into it
• Configure network & wifi (assuming you use supported hardware)
• Create useful aliases
• Install helpful and necessary packages
• Run your own custom provisioning script

Basically, it takes the pain away. No more forgetting how to do certain foundational things. In about 5 minutes, I have a working Raspberry Pi, configured just the way I want, on a brand new SD card.

Have fun. Contribute. Go crazy. Enjoy your pi.

Read More

Easily get your external IP address using the terminal or command line

I want my little Raspberry Pi to "phone home" once in a while so I know where it's at. I wanted a very simple web service that simply regurgitated the requester's IP address in plain text. Nothing more, nothing less.

About to write it myself, I typed in echoip.com and found exactly what I wanted. It's been a good day!

To get your external IP address from the command line or Linux terminal:

$ curl echoip.com

To save it to a file:

$ curl -s echoip.com > myip

Or, if you prefer wget:

$ wget echoip.com --no-cache -qO-

To get super-nerdy and make your life even easier:

$ alias echoip="curl echoip.com"

$ echoip

(You could use any of the variations above for your alias' command.)

In the end, this is the command I added to my Raspberry Pi's crontab (I've changed sensitive parts):

$ curl -s echoip.com | ssh user@mysite.com "cat > /www/mysite/ipaddress.txt"

As a side note, if you want JSON output, there's jsonip.com, or for XML, there's xmlip.com (which actually does XML, JSON, and plain-text, but I feel like the server is a bit slower -- and the homepage doesn't actually serve XML. *facepalm*)

I don't know who made echoip.com, but props to them for choosing the same name I would have. (For some reason, I thought that the output at plainip.com would be... well, plainer.) Memo to self: if echoip ever goes offline, write my own to replace it.

Read More

Using Vagrant and cross-compiling Go (golang)

This is mostly a memo-to-self about how to write Go code in my Mac environment, compile it there for a Linux environment, and run it in a production-like Linux environment using Vagrant.

If using Homebrew...

My Go installation was home-brewed. If you used Homebrew to install Go, you need to make sure you did with the --cross-compile-common flag. If not, no problem, just run these:

$ brew update
$ brew upgrade go --cross-compile-common

Or do --cross-compile-all for all supported platforms.

If not using Homebrew...

If you're not using a Homebrew install of Go, you need to make Go support cross-compilation yourself. First you just need to tell Go to put the package files for the other architecture(s) in its pkg directory.

$ cd /usr/local/go/src

$ sudo GOOS=linux GOARCH=amd64 CGO_ENABLED=0 ./make.bash --no-clean

For GOOS, you can specify linux, darwin (for Mac), windows, and freebsd. For GOARCH, you can specify amd64, 386 (for 32-bit), and arm. As of Go 1.1, cgo is disabled by default, so the CGO_ENABLED part isn't necessary if you're setting it to 0, but to enable it you'll need to set it to 1.

Cross-compile your Go program

Once the make is finished, you can change into your Go program's directory and build or install like this:

$ GOOS=linux GOARCH=amd64 go build -o myprogram.linux64 prog.go

myprogram.linux is what I'm calling the output file in this case to remind me that the binary is for 64-bit Linux systems. For Windows, you'd want to call it something ending in .exe. Finally, prog.go is the name of my source file.

You can also do an install instead of just a build using the same technique.

Set it up with Vagrant

Coming soon.

Read More

Why yes, Go/Golang, I still want to read my CSV file!

UPDATE: This appears to have been fixed and the fix ships with Go 1.2.

I like Go (1.1.1), but how disappointing that in order to read this:

1,O1,,,,0.0000,0.0000,,
2,AP,,,,35.0000,105.0000,,
3,EU,,,,47.0000,8.0000,,
4,AD,,,,42.5000,1.5000,,

I need to tell this code:

csvFile, err := os.Open("/path/myfile.csv")
defer csvFile.Close()
if err != nil {
    panic(err)
}
csvReader := csv.NewReader(csvFile)
for {
    fields, err := csvReader.Read()
    if err == io.EOF {
        break
    } else if err != nil {
        panic(err)
    }
    // ... do stuff ...
}

to allow the last field in any row to be empty:

csvReader.TrailingComma = true

Otherwise:

panic: line 1, column 22: extra delimiter at end of line

Really??

One word: BIZARRE.

Even though my dissatisfaction with Go is quite low, this is probably my biggest disappointment so far. It's an obscure, barely-documented bug feature in the encoding/csv package that will have you scratching your head because your CSV file is perfectly formatted. It's all thanks to this fine logic:

// We don't allow trailing commas.

So by trailing comma, they really mean, empty last fields. Um. Why.

Doesn't Go REQUIRE trailing commas when defining map and struct literals? Why forbid them here? Kind of odd considering it's an engineered language.

A line break or comma in the data would enclose the field in quotes... so a comma at the end of a line must mean that the last field is... empty. That is all.

O little csv.Reader, your name is Reader, not SyntaxChecker. Leave it to the file's author to fix those flaws. But please, just read this properly-formatted CSV file in all its little glory. That's all I ask of you, csv.Reader... that's all I ask...

Read More

Reducing memory usage in Go programs ("golang")

For a particular task, I needed to write a program in Go that reads an 800 MB CSV file, which simply contains a table of numbers. It serves as an index. These numbers represent integers that point to locations in ordered lists stored in other files (an efficient way to index large amounts of data where there is moderate-high duplication).

Loading the table into [][]int used up about 9 GB of memory. But it turns out that most of the six fields only needed one byte of space to store their values, not 8 entire bytes like the int type uses. See Go's documentation on integer types. Very important to know their bounds.

By creating a struct with specialized int8 and int16 and, in just one case, int32 types, and loading the CSV file into []MyStruct instead of [][]int, my program now uses 1/3 the memory: about 3 GB, and it's just as fast. So where one row was using about 48 bytes of space (times 46 million -- that's a lot), each row now uses only about 16 bytes, which is -- that's right -- 1/3 the size.

Read More

Writing a Go ("golang") Web App with nginx, FastCGI, MySQL, JSON

Want to write a web app in Go ("golang") like you write a PHP app? Go is cool since it's kind-of multi-threaded and has some other neat advantages over PHP. I've had fun setting up a small web app in Go. In my case, it's a simple API which accepts JSON input, decodes it into a struct, does a MySQL database request, and returns a JSON response.

Install Go, nginx, and MySQL if not already installed

I will assume you already have these installed and added to your $PATH. If you don't need a database, then don't install MySQL. On my Mac, homebrew was the easiest way to install these -- trust me. (brew install go, brew install nginx, etc.)

If you're using MySQL, then you'll also want to install the Go-MySQL-Driver for Go. Install it simply by doing:

$ go get github.com/go-sql-driver/mysql

Configure nginx/FastCGI

This is actually pretty easy. I assume you already have some experience configuring nginx.conf. (Each install seemingly has different defaults as to the conf file's location, and contents, so I won't go over it here. Mine is in /usr/local/etc/nginx.)

I assume too that you've configured PHP with FastCGI before. If not, you may still understand what's happening.

All you have to do is tell nginx to pass certain requests, or maybe all of them if you wish, to FastCGI on a certain port. Our Go program will have a FastCGI handler listening on that same port. If you need a reference, my entire server { ... } block looks like this:

server {

        listen 80;

        server_name go.dev;

        root /Users/matt/Sites/go;

        index index.html;

        #gzip off;

        #proxy_buffering off;

        location / {

                 try_files $uri $uri/;

        }

        location ~ /app.* {

                include         fastcgi.conf;

                fastcgi_pass    127.0.0.1:9001;

        }

        try_files $uri $uri.html =404;

}

Notice that the server_name is go.dev. I added this to my hosts file with the loopback IP address, 127.0.0.1. So when I type http://go.dev in my browser, it resolves to my own box and nginx receives the request.

Also notice that the fastcgi_pass port is not the default 9000, but is 9001. I did this because I already have php-fpm listening on port 9000. My Go app will listen on 9001.

Further notice that the fastcgi_pass is in a location ~ { ... } block such that any page under /app of go.dev will be redirected to my Go program. You can change this path to whatever you'd like or even replace it with the location / { ... } block above if you want your Go program to be executed out of the root of the site.

Let's see it working

Make a .go file (I'm calling mine app.go) with these contents (though I encourage you to study why/how it's working):

package main

import (

"net"

"net/http"

"net/http/fcgi"

)

type FastCGIServer struct{}

func (s FastCGIServer) ServeHTTP(resp http.ResponseWriter, req *http.Request) {

resp.Write([]byte("<h1>Hello, 世界</h1>\n<p>Behold my Go web app.</p>"))

}

func main() {

listener, _ := net.Listen("tcp", "127.0.0.1:9001")

srv := new(FastCGIServer)

fcgi.Serve(listener, srv)

}

Here's what's happening: The main() function creates a network listener at localhost on port 9001, which is our FastCGI pass-thru port from the nginx config. Then we make a new FastCGIServer and serve requests that pop into that port. The fcgi package has one function: Serve, which blocks and waits for incoming requests. The interface is defined in the net/http package, which is where the ServeHTTP function comes from.

Notice that it receives a ResponseWriter and a Request. From just these, you can write out to your response, and get everything about the incoming request. In this case, all we're doing is writing a simple HTML string.

So, type this in your terminal:

$ go run app.go

Your Go program is now waiting for requests. If you set up nginx.conf like mine, go to http://go.dev/app and you should see, in all its glory:

That was pretty easy!

Now let's receive some simple input from GET/POST

This depends how you want to receive input. If the request was form-URL-encoded (Content-Type: application/x-www-form-urlencoded), like when a form is submitted on a web page, then you can get the form values like this, inside your ServeHTTP function:

fieldValue := req.FormValue("field_name")

This, and more, is documented in the net/http package. Note that fieldValue will be a string unless you convert it to another type yourself. This will nab the field from either POST or GET, but POST takes precedence over GET.

Getting the raw POST request body

But what if you want to get the raw contents of a POST request, like a JSON string? That's also pretty easy. First, make sure a request body exists, then read it. You'll need to import "io/ioutil" since the request Body is a stream. 

if req.Body == nil {

return

}

body, err := ioutil.ReadAll(req.Body)
req.Body.Close()

Decoding JSON

Decoding the JSON is also easy, but it depends on how you want to do it. Don't forget to import "encoding/json". If you know the structure of the JSON or it's pretty simple, you can make a struct to match the field names exactly:

type UserInput struct {

SomeField string,

AnotherField int,

LastOne string

}

Then you can decode the JSON and write out a value, like so:

var inp UserInput

json.Unmarshal(body, &inp)

resp.Write([]byte(m.SomeField))

Simple!

The nice thing is that you don't need to include all the JSON fields that the JSON input has. Any fields you don't include in your struct, or that don't match the JSON input's structure will be ignored.

Encoding (serializing) JSON

Now let's JSON-encode a struct to output.

bytesOfJSON, _ := json.Marshal(myStructOrSlice)

Here I'm throwing away the error, which isn't smart, but for this example, whatever. The encoded string is now contained as a []byte (byte array, or more technically, a byte slice) which conveniently can be passed right into the ResponseWriter's Write function:

resp.Write(bytesOfJSON)

Tada!

A simple MySQL database request

Now let's do something with the database. Assuming you ran the "go get" line at the very beginning of this article and have the driver's package installed, import these two packages:

"database/sql"

_ "github.com/go-sql-driver/mysql"

I chose the Go SQL Driver over MyMySQL because the Go SQL Driver uses the native database/sql package and is a little newer. But both packages are excellent and work without any real problems.

Making the connection (under the hood, it doesn't actually make the connection until needed, from what I understand) is simple:

conn, err := sql.Open("mysql", "/test")

defer conn.Close()

if err != nil {

fmt.Println("Oh noez, could not connect to database")

return

}

Let me explain the first line. The first argument is the type of database, in our case, "mysql" will do. The second argument is the connection string, which, if you're familiar with PEAR DB, works almost the same way. In my case, I simply specified "/test" because the default user is "root" and I have no password on my dev machine's MySQL server, so those all go away because of defaults, and the name of my database is "test". See the docs for a brief overview of the format for the connection string (it's really quite easy).

The defer line ensures that the connection is closed when we're through with it.

Then to query a row and get a single result, I do this:

var zipcode string

sqlerr := conn.QueryRow("SELECT ZipCode FROM Cities WHERE CityName=? LIMIT 1", "Las Vegas").Scan(&zipcode)

switch {

case sqlerr == sql.ErrNoRows:

    fmt.Printf("No rows")

case sqlerr != nil:

    fmt.Println(sqlerr)

default:

    fmt.Printf("ZIP code is %s\n", zipcode)

}

Now, this only gets one ZIP code for Las Vegas, but there are actually many. How do we get the rest of them?

Easy:

var zipcodes []string

rows, err := db.Query("SELECT ZipCode FROM Cities WHERE CityName=?", "Las Vegas")

if err != nil {
fmt.Println(err)
return
}

for rows.Next() {
var zip string
if err := rows.Scan(&zip); err != nil {
fmt.Println(err)
return
}
zipcodes = append(zipcodes, zip)
}

if err := rows.Err(); err != nil {
fmt.Println(err)
return
}

Ahhhh... much better. Writing that out to the response (as a JSON array, of course) gives me a list of ZIP codes in Las Vegas:

["87701","87745","89044","89054","89101","89102","89103","89104","89105","89106",

"89107","89108","89109","89110","89111","89112","89113","89114","89115","89116",

"89117","89118","89119","89120","89121","89122","89123","89124","89125","89126",

"89127","89128","89129","89130","89131","89132","89133","89134","89135","89136",

"89137","89138","89139","89140","89141","89142","89143","89144","89145","89146",

"89147","89148","89149","89150","89151","89152","89153","89154","89155","89156",

"89157","89158","89159","89160","89161","89162","89163","89164","89165","89166",

"89169","89170","89173","89177","89178","89179","89180","89183","89185","89191",

"89193","89195","89199"]

So... now what?

Well, now that we can read requests, send responses, use MySQL, encode/decode JSON, and all through nginx and FastCGI which is available even on shared hosting, I'd say the possibilities are nearly endless. Ready? Set... Go!

Read More