Fix Home Needs

  • Subscribe to our RSS feed.
  • Twitter
  • StumbleUpon
  • Reddit
  • Facebook
  • Digg

Friday, 24 May 2013

Writing a Go ("golang") Web App with nginx, FastCGI, MySQL, JSON

Posted on 09:49 by Unknown
Want to write a web app in Go ("golang") like you write a PHP app? Go is cool since it's kind-of multi-threaded and has some other neat advantages over PHP. I've had fun setting up a small web app in Go. In my case, it's a simple API which accepts JSON input, decodes it into a struct, does a MySQL database request, and returns a JSON response.

Install Go, nginx, and MySQL if not already installed

I will assume you already have these installed and added to your $PATH. If you don't need a database, then don't install MySQL. On my Mac, homebrew was the easiest way to install these -- trust me. (brew install go, brew install nginx, etc.)

If you're using MySQL, then you'll also want to install the Go-MySQL-Driver for Go. Install it simply by doing:

$ go get github.com/go-sql-driver/mysql

Configure nginx/FastCGI

This is actually pretty easy. I assume you already have some experience configuring nginx.conf. (Each install seemingly has different defaults as to the conf file's location, and contents, so I won't go over it here. Mine is in /usr/local/etc/nginx.)

I assume too that you've configured PHP with FastCGI before. If not, you may still understand what's happening.

All you have to do is tell nginx to pass certain requests, or maybe all of them if you wish, to FastCGI on a certain port. Our Go program will have a FastCGI handler listening on that same port. If you need a reference, my entire server { ... } block looks like this:

server {
        listen 80;
        server_name go.dev;
        root /Users/matt/Sites/go;
        index index.html;
        #gzip off;
        #proxy_buffering off;

        location / {
                 try_files $uri $uri/;
        }

        location ~ /app.* {
                include         fastcgi.conf;
                fastcgi_pass    127.0.0.1:9001;
        }

        try_files $uri $uri.html =404;
}

Notice that the server_name is go.dev. I added this to my hosts file with the loopback IP address, 127.0.0.1. So when I type http://go.dev in my browser, it resolves to my own box and nginx receives the request.

Also notice that the fastcgi_pass port is not the default 9000, but is 9001. I did this because I already have php-fpm listening on port 9000. My Go app will listen on 9001.

Further notice that the fastcgi_pass is in a location ~ { ... } block such that any page under /app of go.dev will be redirected to my Go program. You can change this path to whatever you'd like or even replace it with the location / { ... } block above if you want your Go program to be executed out of the root of the site.

Let's see it working

Make a .go file (I'm calling mine app.go) with these contents (though I encourage you to study why/how it's working):

package main

import (
"net"
"net/http"
"net/http/fcgi"
)

type FastCGIServer struct{}

func (s FastCGIServer) ServeHTTP(resp http.ResponseWriter, req *http.Request) {
resp.Write([]byte("<h1>Hello, 世界</h1>\n<p>Behold my Go web app.</p>"))
}

func main() {
listener, _ := net.Listen("tcp", "127.0.0.1:9001")
srv := new(FastCGIServer)
fcgi.Serve(listener, srv)
}

Here's what's happening: The main() function creates a network listener at localhost on port 9001, which is our FastCGI pass-thru port from the nginx config. Then we make a new FastCGIServer and serve requests that pop into that port. The fcgi package has one function: Serve, which blocks and waits for incoming requests. The interface is defined in the net/http package, which is where the ServeHTTP function comes from.

Notice that it receives a ResponseWriter and a Request. From just these, you can write out to your response, and get everything about the incoming request. In this case, all we're doing is writing a simple HTML string.

So, type this in your terminal:

$ go run app.go

Your Go program is now waiting for requests. If you set up nginx.conf like mine, go to http://go.dev/app and you should see, in all its glory:


That was pretty easy!

Now let's receive some simple input from GET/POST

This depends how you want to receive input. If the request was form-URL-encoded (Content-Type: application/x-www-form-urlencoded), like when a form is submitted on a web page, then you can get the form values like this, inside your ServeHTTP function:

fieldValue := req.FormValue("field_name")

This, and more, is documented in the net/http package. Note that fieldValue will be a string unless you convert it to another type yourself. This will nab the field from either POST or GET, but POST takes precedence over GET.

Getting the raw POST request body

But what if you want to get the raw contents of a POST request, like a JSON string? That's also pretty easy. First, make sure a request body exists, then read it. You'll need to import "io/ioutil" since the request Body is a stream. 

if req.Body == nil {
return
}
body, err := ioutil.ReadAll(req.Body)
req.Body.Close()

Decoding JSON

Decoding the JSON is also easy, but it depends on how you want to do it. Don't forget to import "encoding/json". If you know the structure of the JSON or it's pretty simple, you can make a struct to match the field names exactly:

type UserInput struct {
SomeField string,
AnotherField int,
LastOne string
}

Then you can decode the JSON and write out a value, like so:

var inp UserInput
json.Unmarshal(body, &inp)
resp.Write([]byte(m.SomeField))

Simple!

The nice thing is that you don't need to include all the JSON fields that the JSON input has. Any fields you don't include in your struct, or that don't match the JSON input's structure will be ignored.

Encoding (serializing) JSON

Now let's JSON-encode a struct to output.

bytesOfJSON, _ := json.Marshal(myStructOrSlice)

Here I'm throwing away the error, which isn't smart, but for this example, whatever. The encoded string is now contained as a []byte (byte array, or more technically, a byte slice) which conveniently can be passed right into the ResponseWriter's Write function:

resp.Write(bytesOfJSON)

Tada!

A simple MySQL database request

Now let's do something with the database. Assuming you ran the "go get" line at the very beginning of this article and have the driver's package installed, import these two packages:

"database/sql"
_ "github.com/go-sql-driver/mysql"

I chose the Go SQL Driver over MyMySQL because the Go SQL Driver uses the native database/sql package and is a little newer. But both packages are excellent and work without any real problems.

Making the connection (under the hood, it doesn't actually make the connection until needed, from what I understand) is simple:

conn, err := sql.Open("mysql", "/test")
defer conn.Close()

if err != nil {
fmt.Println("Oh noez, could not connect to database")
return
}

Let me explain the first line. The first argument is the type of database, in our case, "mysql" will do. The second argument is the connection string, which, if you're familiar with PEAR DB, works almost the same way. In my case, I simply specified "/test" because the default user is "root" and I have no password on my dev machine's MySQL server, so those all go away because of defaults, and the name of my database is "test". See the docs for a brief overview of the format for the connection string (it's really quite easy).

The defer line ensures that the connection is closed when we're through with it.

Then to query a row and get a single result, I do this:

var zipcode string
sqlerr := conn.QueryRow("SELECT ZipCode FROM Cities WHERE CityName=? LIMIT 1", "Las Vegas").Scan(&zipcode)
switch {
case sqlerr == sql.ErrNoRows:
    fmt.Printf("No rows")
case sqlerr != nil:
    fmt.Println(sqlerr)
default:
    fmt.Printf("ZIP code is %s\n", zipcode)
}

Now, this only gets one ZIP code for Las Vegas, but there are actually many. How do we get the rest of them?

Easy:

var zipcodes []string

rows, err := db.Query("SELECT ZipCode FROM Cities WHERE CityName=?", "Las Vegas")

if err != nil {
fmt.Println(err)
return
}

for rows.Next() {
var zip string
if err := rows.Scan(&zip); err != nil {
fmt.Println(err)
return
}
zipcodes = append(zipcodes, zip)
}

if err := rows.Err(); err != nil {
fmt.Println(err)
return
}

Ahhhh... much better. Writing that out to the response (as a JSON array, of course) gives me a list of ZIP codes in Las Vegas:


["87701","87745","89044","89054","89101","89102","89103","89104","89105","89106",
"89107","89108","89109","89110","89111","89112","89113","89114","89115","89116",
"89117","89118","89119","89120","89121","89122","89123","89124","89125","89126",
"89127","89128","89129","89130","89131","89132","89133","89134","89135","89136",
"89137","89138","89139","89140","89141","89142","89143","89144","89145","89146",
"89147","89148","89149","89150","89151","89152","89153","89154","89155","89156",
"89157","89158","89159","89160","89161","89162","89163","89164","89165","89166",
"89169","89170","89173","89177","89178","89179","89180","89183","89185","89191",
"89193","89195","89199"]

So... now what?

Well, now that we can read requests, send responses, use MySQL, encode/decode JSON, and all through nginx and FastCGI which is available even on shared hosting, I'd say the possibilities are nearly endless. Ready? Set... Go!
Read More
Posted in fastcgi, fcgi, go, golang, mac, nginx, osx | No comments

Wednesday, 10 April 2013

Install PECL extensions with Homebrewed PHP on Mac OS X Mountain Lion

Posted on 14:36 by Unknown
I needed to use the HTTPRequest class that comes as a PECL extension to PHP. This was easier than I thought. If you followed my instructions for installing PHP with Homebrew, then installing the PECL extensions should be even easier. Here's how:

Open up the Terminal and do:

$ pecl list-all

just as a sanity check to make sure you have PECL installed from your Homebrew installation of PHP. You will see a list of all available PECL packages, and, potentially after some "Warnings" from bugs in the PEAR code, you'll notice that "pecl/pecl_http" is in the list.

Before you can install anything with pecl, you need autoconf. It helps to build in the environment with which we're working. Just do:

$ brew install autoconf

Then install your favorite PECL extension (in my case, pecl_http):

$ pecl install pecl_http

Then a whole bunch of code garbage will fly past the screen. In a minute or two, some config options will appear. Press "return" to accept the defaults in brackets. When it's finished, you'll know.

If you got a fatal error during install, try running under sudo. If you still get an error, wait a day or two and maybe they'll push a bug fix?

Then just restart your web service (I just reboot my computer) and you're done!
Read More
Posted in homebrew, install, mac, mountain lion, osx, pecl, php, terminal | No comments

Friday, 5 April 2013

Running Inkscape in Mac OS X Mountain Lion (finding X11)

Posted on 13:10 by Unknown
OS X 10.8 (Mountain Lion) doesn't come with X11 (an X.org window server) installed like previous versions do. But getting Inkscape to run is still very easy.

Download the latest XQuartz and install it.

Next time you run Inkscape, it will ask you, "Where is X11?" It probably won't be in that list, so browse for it in your "Applications" folder, then go inside "Utilities" and choose XQuartz.

Then give it maybe several minutes to run the first time. And there you have it!
Read More
Posted in inkscape, install, mountain lion, osx | No comments

Tuesday, 12 March 2013

PHP output buffering on nginx

Posted on 19:48 by Unknown
Simple trick: if you're trying to use output buffering with PHP on nginx, and it should be working, but the whole response still comes down all at once to the client, try disabling gzip compression:

nginx.conf

gzip off;

That should do it! Sure, your "fast" pages might become a little slower, but your slow pages will appear to be loading instead of hanging.

(This works because, of course, nginx can't compress the output until it's all ready to go first!)
Read More
Posted in nginx, php | No comments

Thursday, 7 March 2013

Install nginx / PHP / MySQL on Mac OS X Mountain Lion with Homebrew

Posted on 08:35 by Unknown
Last time I wiped my Macbook Pro, I used Macports to install my web development environment. Doing it that way was really hard compared to using Homebrew. I now fully recommend Homebrew for all Mac package management needs. This is much easier than the Macports way. Trust me.

Here's how to install nginx, PHP, and MySQL using Homebrew on your Mac. It's actually quite easy. I did it on Mountain Lion (10.8) but it probably works for Lion too. I followed the tutorial here on EZUnix.org, but my version fixes some typos and explains some steps along the way.

Disclaimer: This is new to me; I'm not an expert. I didn't encounter any problems, and I did this from a clean install of OS X Mountain Lion. If you encounter any errors, I may or may not be able to help...



Estimated Time: 10-20 minutes



Got Command Line Developer Tools?


You're gonna need them. Finally, Apple provides the command line tools without needing to install the nearly-2GB-Xcode from the App Store. Go to their Developers Downloads page and download the latest "Command Line Tools" for your version of OS X, then install them.


Install Homebrew


In case you haven't already, install Homebrew by following the instructions at the bottom of this page.

Homebrew's most legit PHP "tap" (package source) is by Jose Gonzalez. Make sure to install it:

$ brew tap josegonzalez/homebrew-php

We also need a tap for a PHP 5.4 dependency, zlib:

$ brew tap homebrew/dupes

Install MySQL

$ brew install mysql

It'll chew on that for a few minutes, then we need to get it to run as our user account:

$ unset TMPDIR
$ mysql_install_db --verbose --user=`whoami` --basedir="$(brew --prefix mysql)" --datadir=/usr/local/var/mysql --tmpdir=/tmp

I got a "Warning" during this operation, and while I don't think it's critical, I did this and things have seemed to work fine... if you got a warning during the last step, then you could do this:

$ sudo mv /usr/local/opt/mysql/my-new.cnf /usr/local/opt/mysql/my.cnf

Then, to launch MySQL at startup:

$ cp `brew --prefix mysql`/homebrew.mxcl.mysql.plist ~/Library/LaunchAgents/
$ launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist

Done! Next: the web server.


Install nginx

$ brew install nginx

Let that stew, then run these commands to have nginx run as root at startup (so we can listen on port 80, the default, instead of 8080 which is less convenient for development):

$ sudo cp `brew --prefix nginx`/homebrew.mxcl.nginx.plist /Library/LaunchDaemons/
$ sudo sed -i -e 's/`whoami`/root/g' `brew --prefix nginx`/homebrew.mxcl.nginx.plist


(Okay, to be honest, this didn't work for me to load nginx right away on start; I had to edit the /Library/LaunchDaemons/homebrew.mxcl.nginx.plist file and remove the two lines that specify the UserName key and value (one line specifies the key, the other the value). Then it worked.)

Then create a log file... this allows us to view server logs in the Mac Console, which is really convenient, but isn't required:

$ sudo mkdir /var/log/nginx/

(Don't forget to tell nginx to put the log file there in nginx.conf: error_log  /var/log/nginx/error.log;)

Done! Next up: PHP.


Install PHP

$ brew install --without-apache --with-fpm --with-mysql php54

Make sure to change "php54" to whatever version you want. At time of writing, PHP 5.4 is the latest stable, but PHP 5.5 is in alpha. I assume 5.5 would be php55, etc. Be sure to adjust any following commands with the proper version number.


Quick note: Yes, OS X does come with PHP pre-installed. But we don't want to use that. We need an install we can use with nginx and FastCGI Process Manager (fpm). Plus, we want the latest version, and I'm just not that into compiling from source.

To run php-fpm at startup:

$ sudo cp `brew --prefix php54`/homebrew-php.josegonzalez.php54.plist  /Library/LaunchAgents/
$ sudo launchctl load -w /Library/LaunchAgents/homebrew-php.josegonzalez.php54.plist

Done! Next up: configuration.



Finishing up

I want all php commands to be using the latest version, not the default PHP binary. So I use this little trick to create a symlink from the default PHP binary to the new one... I do this for both php and php-fpm. If you're confused about which versions are where, use the "whereis" command, like: "whereis php".


$ php-fpm -v
$ sudo mv /usr/sbin/php-fpm /usr/sbin/php-fpm.bak
$ sudo ln -s /usr/local/Cellar/php54/5.4.11/sbin/php-fpm /usr/sbin/php-fpm
$ php-fpm -v

Notice that the version went from 5.3 to 5.4 (in my case). Now for the php binary:

$ php -v
$ sudo mv /usr/bin/php /usr/bin/php.bak
$ sudo ln -s /usr/local/bin/php /usr/bin/php
$ php -v

I also added /usr/local/sbin to the PATH by adding that directory to the /etc/paths file, then restarting Terminal. You can see your current PATH by typing echo $PATH.

Important config files:

/usr/local/etc/nginx/nginx.conf
/usr/local/etc/php/5.4/php.ini
/usr/local/etc/nginx/fastcgi_params

You'll probably want to change these for your

The nice thing about Homebrew installations is that you generally don't need sudo to use or manage these services, since they're in /usr/local.

Alright. Well that did it for me. Enjoy your new dev environment!

You can stop nginx with nginx -s stop, and start it again with just nginx. You can also just reload the conf file with nginx -s reload.

I installed MySQL Workbench and was able to make a connection to the localhost MySQL server by adding a connection to host "localhost" with no password. The only thing I typed was that hostname and everything worked like a charm.

I did use my nginx.conf file from my previous install; you can view a sample conf file if you need it, in my other post about using Macports to do this (link at top of this post).
Read More
Posted in development, mac, mountain lion, osx, php, terminal | No comments

Saturday, 16 February 2013

PHP's PECL extension (for HttpRequest) worked on website but not command line (CLI)?

Posted on 16:44 by Unknown
I'm running PHP on my Mac under nginx and FastCGI... and that's great and fine. I used Macports to set that all up.

Well, Macs have PHP in another, default location: /usr/bin/php, not Macport's /opt/local/bin/php. When I installed the PECL extensions using Macports (after installing PHP), it installed them to the PHP at /opt/local/bin/php, not /usr/bin/php.

I have a PHP script that makes HTTP requests, and it may be a long process, so I spin up PHP on the command-line to do it in the background. Took me forever to figure out that that PHP binary was different from the one used by nginx when I loaded up my dev site.

The CLI version of PHP which I was running for this didn't have PECL installed. A quick, dirty way to fix this:

  1. sudo mv /usr/bin/php /usr/bin/php.bak
  2. sudo ls -s /opt/local/bin/php /usr/bin/php

That's right! Make php a symlink! Why? Well, I'm not sure. I tried simply doing the "mv" to kind of "hide" the binary from sh, but it was giving me "php - command not found" errors, even though /opt/local/bin is was in the PATH. Why didn't it look there? I have no idea. I'm not very proficient with unix stuff. But the symlink was enough to trick it and it's working great now.
Read More
Posted in cli, mac, pecl, php | No comments

Tuesday, 27 November 2012

How to remove a word from the dictionary (Chrome, Mac OS X)

Posted on 10:51 by Unknown
This trick works for any program on the Mac which uses the default dictionary (the same one that TextEdit, Mail, and the other built-in programs use). So when I accidentally added a misspelled word to the dictionary in Google Chrome, I wanted to remove it. Here's how:

  1. Open TextEdit
  2. Type the misspelled word
  3. Right-click it and click "Unlearn Spelling"
Done! Works across-the-board on Mac OS 10.5 (I'm pretty sure), and higher -- I am running Mountain Lion.
Read More
Posted in chrome, mac, osx | No comments
Newer Posts Older Posts Home
Subscribe to: Posts (Atom)

Popular Posts

  • How to take FrontRunner from Provo to SLC airport
    I see this question a lot: how do I get from Provo or Orem to the SLC International Airport entirely by train (UTA FrontRunner/Trax)? Here I...
  • Behavior-driven testing in Go with GoConvey (BDD in "golang")
    First: the built-in Go testing tools Few things bring sweeter peace to the soul than making changes to Go code, then: $ go test ... PASS ok ...
  • Fix the Home and End keys on Mac OS X (Mountain Lion)
    If you use a keyboard that's not designed specifically for Macs, you probably are familiar with the annoying mapping of the Home and End...
  • Why yes, Go/Golang, I still want to read my CSV file!
    UPDATE: This appears to have been fixed and the fix  ships with   Go 1.2 . I like Go (1.1.1), but how disappointing that in order to read ...
  • Installing nginx / PHP / MySQL on Mac OS X Mountain Lion
    ** Update: See a quicker way to do this using Homebrew (this method uses Macports, and it's considerably more difficult). ** ... are yo...
  • Using Vagrant and cross-compiling Go (golang)
    This is mostly a memo-to-self about how to write Go code in my Mac environment, compile it there for a Linux environment, and run it in a pr...
  • Automatically make a Raspberry Pi with wifi support
    Okay, I love my Raspberry Pi, but setting it up just the way I want got so involved I was afraid I couldn't do it again if I had to. So ...
  • Install nginx / PHP / MySQL on Mac OS X Mountain Lion with Homebrew
    Last time I wiped my Macbook Pro, I used Macports to install my web development environment . Doing it that way was really hard compared to ...
  • Printing to BYU campus printers without extra software (Mac OS X)
    Macs come with a print server installed by default. This means you don't need to install any drivers or software to print to the BYU ca...
  • External hard drive backups while you sleep
    On most modern computers, there's an energy saver preference which will shut down your hard disks when the computer is idle or in "...

Categories

  • backup
  • bdd
  • byu
  • chrome
  • cli
  • command line
  • commute
  • compile
  • cross-compile
  • csv
  • development
  • dns
  • domain name
  • fastcgi
  • fcgi
  • go
  • golang
  • homebrew
  • inkscape
  • install
  • ip
  • ip address
  • ipaddress
  • itunes
  • javascript
  • keybinding
  • linux
  • mac
  • mountain lion
  • mysql
  • nginx
  • optimization
  • osx
  • parsing
  • pecl
  • php
  • printing
  • raspberry pi
  • security
  • ssd
  • terminal
  • testing
  • transportation
  • unit tests
  • vagrant

Blog Archive

  • ▼  2013 (16)
    • ▼  November (1)
      • How to take FrontRunner from Provo to SLC airport
    • ►  October (1)
    • ►  September (2)
    • ►  August (2)
    • ►  July (3)
    • ►  June (1)
    • ►  May (1)
    • ►  April (2)
    • ►  March (2)
    • ►  February (1)
  • ►  2012 (8)
    • ►  November (1)
    • ►  October (1)
    • ►  September (6)
Powered by Blogger.

About Me

Unknown
View my complete profile