Fix Home Needs

  • Subscribe to our RSS feed.
  • Twitter
  • StumbleUpon
  • Reddit
  • Facebook
  • Digg
Showing posts with label nginx. Show all posts
Showing posts with label nginx. Show all posts

Friday, 24 May 2013

Writing a Go ("golang") Web App with nginx, FastCGI, MySQL, JSON

Posted on 09:49 by Unknown
Want to write a web app in Go ("golang") like you write a PHP app? Go is cool since it's kind-of multi-threaded and has some other neat advantages over PHP. I've had fun setting up a small web app in Go. In my case, it's a simple API which accepts JSON input, decodes it into a struct, does a MySQL database request, and returns a JSON response.

Install Go, nginx, and MySQL if not already installed

I will assume you already have these installed and added to your $PATH. If you don't need a database, then don't install MySQL. On my Mac, homebrew was the easiest way to install these -- trust me. (brew install go, brew install nginx, etc.)

If you're using MySQL, then you'll also want to install the Go-MySQL-Driver for Go. Install it simply by doing:

$ go get github.com/go-sql-driver/mysql

Configure nginx/FastCGI

This is actually pretty easy. I assume you already have some experience configuring nginx.conf. (Each install seemingly has different defaults as to the conf file's location, and contents, so I won't go over it here. Mine is in /usr/local/etc/nginx.)

I assume too that you've configured PHP with FastCGI before. If not, you may still understand what's happening.

All you have to do is tell nginx to pass certain requests, or maybe all of them if you wish, to FastCGI on a certain port. Our Go program will have a FastCGI handler listening on that same port. If you need a reference, my entire server { ... } block looks like this:

server {
        listen 80;
        server_name go.dev;
        root /Users/matt/Sites/go;
        index index.html;
        #gzip off;
        #proxy_buffering off;

        location / {
                 try_files $uri $uri/;
        }

        location ~ /app.* {
                include         fastcgi.conf;
                fastcgi_pass    127.0.0.1:9001;
        }

        try_files $uri $uri.html =404;
}

Notice that the server_name is go.dev. I added this to my hosts file with the loopback IP address, 127.0.0.1. So when I type http://go.dev in my browser, it resolves to my own box and nginx receives the request.

Also notice that the fastcgi_pass port is not the default 9000, but is 9001. I did this because I already have php-fpm listening on port 9000. My Go app will listen on 9001.

Further notice that the fastcgi_pass is in a location ~ { ... } block such that any page under /app of go.dev will be redirected to my Go program. You can change this path to whatever you'd like or even replace it with the location / { ... } block above if you want your Go program to be executed out of the root of the site.

Let's see it working

Make a .go file (I'm calling mine app.go) with these contents (though I encourage you to study why/how it's working):

package main

import (
"net"
"net/http"
"net/http/fcgi"
)

type FastCGIServer struct{}

func (s FastCGIServer) ServeHTTP(resp http.ResponseWriter, req *http.Request) {
resp.Write([]byte("<h1>Hello, 世界</h1>\n<p>Behold my Go web app.</p>"))
}

func main() {
listener, _ := net.Listen("tcp", "127.0.0.1:9001")
srv := new(FastCGIServer)
fcgi.Serve(listener, srv)
}

Here's what's happening: The main() function creates a network listener at localhost on port 9001, which is our FastCGI pass-thru port from the nginx config. Then we make a new FastCGIServer and serve requests that pop into that port. The fcgi package has one function: Serve, which blocks and waits for incoming requests. The interface is defined in the net/http package, which is where the ServeHTTP function comes from.

Notice that it receives a ResponseWriter and a Request. From just these, you can write out to your response, and get everything about the incoming request. In this case, all we're doing is writing a simple HTML string.

So, type this in your terminal:

$ go run app.go

Your Go program is now waiting for requests. If you set up nginx.conf like mine, go to http://go.dev/app and you should see, in all its glory:


That was pretty easy!

Now let's receive some simple input from GET/POST

This depends how you want to receive input. If the request was form-URL-encoded (Content-Type: application/x-www-form-urlencoded), like when a form is submitted on a web page, then you can get the form values like this, inside your ServeHTTP function:

fieldValue := req.FormValue("field_name")

This, and more, is documented in the net/http package. Note that fieldValue will be a string unless you convert it to another type yourself. This will nab the field from either POST or GET, but POST takes precedence over GET.

Getting the raw POST request body

But what if you want to get the raw contents of a POST request, like a JSON string? That's also pretty easy. First, make sure a request body exists, then read it. You'll need to import "io/ioutil" since the request Body is a stream. 

if req.Body == nil {
return
}
body, err := ioutil.ReadAll(req.Body)
req.Body.Close()

Decoding JSON

Decoding the JSON is also easy, but it depends on how you want to do it. Don't forget to import "encoding/json". If you know the structure of the JSON or it's pretty simple, you can make a struct to match the field names exactly:

type UserInput struct {
SomeField string,
AnotherField int,
LastOne string
}

Then you can decode the JSON and write out a value, like so:

var inp UserInput
json.Unmarshal(body, &inp)
resp.Write([]byte(m.SomeField))

Simple!

The nice thing is that you don't need to include all the JSON fields that the JSON input has. Any fields you don't include in your struct, or that don't match the JSON input's structure will be ignored.

Encoding (serializing) JSON

Now let's JSON-encode a struct to output.

bytesOfJSON, _ := json.Marshal(myStructOrSlice)

Here I'm throwing away the error, which isn't smart, but for this example, whatever. The encoded string is now contained as a []byte (byte array, or more technically, a byte slice) which conveniently can be passed right into the ResponseWriter's Write function:

resp.Write(bytesOfJSON)

Tada!

A simple MySQL database request

Now let's do something with the database. Assuming you ran the "go get" line at the very beginning of this article and have the driver's package installed, import these two packages:

"database/sql"
_ "github.com/go-sql-driver/mysql"

I chose the Go SQL Driver over MyMySQL because the Go SQL Driver uses the native database/sql package and is a little newer. But both packages are excellent and work without any real problems.

Making the connection (under the hood, it doesn't actually make the connection until needed, from what I understand) is simple:

conn, err := sql.Open("mysql", "/test")
defer conn.Close()

if err != nil {
fmt.Println("Oh noez, could not connect to database")
return
}

Let me explain the first line. The first argument is the type of database, in our case, "mysql" will do. The second argument is the connection string, which, if you're familiar with PEAR DB, works almost the same way. In my case, I simply specified "/test" because the default user is "root" and I have no password on my dev machine's MySQL server, so those all go away because of defaults, and the name of my database is "test". See the docs for a brief overview of the format for the connection string (it's really quite easy).

The defer line ensures that the connection is closed when we're through with it.

Then to query a row and get a single result, I do this:

var zipcode string
sqlerr := conn.QueryRow("SELECT ZipCode FROM Cities WHERE CityName=? LIMIT 1", "Las Vegas").Scan(&zipcode)
switch {
case sqlerr == sql.ErrNoRows:
    fmt.Printf("No rows")
case sqlerr != nil:
    fmt.Println(sqlerr)
default:
    fmt.Printf("ZIP code is %s\n", zipcode)
}

Now, this only gets one ZIP code for Las Vegas, but there are actually many. How do we get the rest of them?

Easy:

var zipcodes []string

rows, err := db.Query("SELECT ZipCode FROM Cities WHERE CityName=?", "Las Vegas")

if err != nil {
fmt.Println(err)
return
}

for rows.Next() {
var zip string
if err := rows.Scan(&zip); err != nil {
fmt.Println(err)
return
}
zipcodes = append(zipcodes, zip)
}

if err := rows.Err(); err != nil {
fmt.Println(err)
return
}

Ahhhh... much better. Writing that out to the response (as a JSON array, of course) gives me a list of ZIP codes in Las Vegas:


["87701","87745","89044","89054","89101","89102","89103","89104","89105","89106",
"89107","89108","89109","89110","89111","89112","89113","89114","89115","89116",
"89117","89118","89119","89120","89121","89122","89123","89124","89125","89126",
"89127","89128","89129","89130","89131","89132","89133","89134","89135","89136",
"89137","89138","89139","89140","89141","89142","89143","89144","89145","89146",
"89147","89148","89149","89150","89151","89152","89153","89154","89155","89156",
"89157","89158","89159","89160","89161","89162","89163","89164","89165","89166",
"89169","89170","89173","89177","89178","89179","89180","89183","89185","89191",
"89193","89195","89199"]

So... now what?

Well, now that we can read requests, send responses, use MySQL, encode/decode JSON, and all through nginx and FastCGI which is available even on shared hosting, I'd say the possibilities are nearly endless. Ready? Set... Go!
Read More
Posted in fastcgi, fcgi, go, golang, mac, nginx, osx | No comments

Tuesday, 12 March 2013

PHP output buffering on nginx

Posted on 19:48 by Unknown
Simple trick: if you're trying to use output buffering with PHP on nginx, and it should be working, but the whole response still comes down all at once to the client, try disabling gzip compression:

nginx.conf

gzip off;

That should do it! Sure, your "fast" pages might become a little slower, but your slow pages will appear to be loading instead of hanging.

(This works because, of course, nginx can't compress the output until it's all ready to go first!)
Read More
Posted in nginx, php | No comments

Tuesday, 25 September 2012

Installing nginx / PHP / MySQL on Mac OS X Mountain Lion

Posted on 15:31 by Unknown

** Update: See a quicker way to do this using Homebrew (this method uses Macports, and it's considerably more difficult). **







... are you sure you want to use Macports to do this? (See the link above for Homebrew instructions instead!) ...








... okay, fine, continue on if you're sure ....





Introduction

Even though OS X comes with Apache server pre-installed, our production environment at work is nginx. While nginx has downloads for Windows and Linux, there's no official mac port. Fortunately, there's Macports.

Okay, true: Macports isn't required to install nginx on a Mac. It's said that installing nginx without Macports is as easy as wget, ./configure, make, and make install. Unfortunately, I usually run into problems when compiling and installing from source. I'm definitely no Macports junkie, but I am too pressed for time to debug someone else's huge code base to compile on my machine.

That said, here's the lazy way to install a complete web development environment on your Mac. These instructions are designed for a clean install of Mountain Lion (mine is 10.8.2). It's not as bad as it looks, I promise. Except the "prep" section, skip anything you don't want installed. And hey, you'll learn a lot in the process (like I did). It's actually pretty systematic...

Prep

Install Xcode. Sorry, it's gotta happen. Macports needs it. While it's downloading, hop over to Macports.org and get the latest pkg for your system. Or run: wget https://distfiles.macports.org/MacPorts/MacPorts-2.1.2-10.8-MountainLion.pkg from the terminal.

Run Xcode and accept the license agreement. Then go to Preferences --> Downloads and install the "Command Line Tools." It's another 100+ MB to download, but think of this way: you'll be prepared with pretty much every Apple dev tool you'll ever need after today.

Ensure other web servers are off and disabled. This is a great activity while you wait for Xcode to install. On Mountain Lion, you have to use the command line to enable the default Apache server, so never-mind if you've never done that. Lion and prior, it's easily enabled in Sharing preferences ("Web"). Ensure these are off by running ps aux | grep apache or going to http://localhost in your browser.

Install Macports. Just run the .pkg file you downloaded. Make sure you've completed the previous steps first. By default, Macports and all ports will go into /opt/local. The readme file you see at install is super-helpful to know. (By the way, I had a hard time finding that exact readme file anywhere, so I'm hosting my own copy of it on my Raspberry Pi sitting on my windowsill, just for fun.) To be sure, run sudo port selfupdate when it's done installing to be up-to-date.


Install

nginx


Run sudo port install nginx to install nginx. Twiddle thumbs... and at the end of the install, you'll be shown a command that causes it to run at system start: sudo port load nginx.

Configure nginx. Start by setting the default config files:
  • cd /opt/local/etc/nginx
  • sudo cp nginx.conf.default nginx.conf
  • sudo cp mime.types.default mime.types
  • sudo nginx -s reload
  • Load http://localhost in your browser to see that it's working. You're now done installing nginx.

MySQL

Type: sudo port install mysql5-server and twiddle your thumbs again. When it's done, it gives you a command to have it run at start-up if you wish: sudo port load mysql5-server. For some reason, I had to restart my computer to get MySQL to start, but you can do that later. And the default, "root" user password is simply blank which is nice for dev environments.


PHP with FastCGI


Install PHP 5 and FastCGI. The command is long because you have to include all the extensions / helper libraries you want it compiled with. You can pick and choose, but be sure to include some essential ones like mysql, http, etc. I did it in two pieces like this:
  • sudo port install php5 +fastcgi fcgi
  • sudo port install php5-openssl php5-curl php5-gd php5-iconv php5-http php5-mcrypt php5-xdebug php5-mysql
At this point, your development tools and environment should be all there, and now we just need to configure them.

Configure nginx

Edit nginx.conf. In order to develop multiple sites on my Mac, I prefer to use a separate directory in the nginx folder which contains the config files for each domain I'm developing on. I call this "sites-enabled," located in /opt/local/etc/nginx. You can use the default conf file if you want, but you'll want to make sure a line like this appears somewhere before the final curly brace:

include sites-enabled/*;

This pulls in the configuration files from the sites-enabled folder.

Configure nginx site config files. For each domain/site you are developing, create a file inside /opt/local/etc/nginx/sites-enabled (make that directory if you want) called something like "mysite.dev.conf." A basic config file looks like this:

server {
listen 80;
server_name mysite.dev;
root /Users/matt/Sites/whatever/.../path/no-trailing-slash;

location / {
try_files $uri $uri/ /index.html;
}

try_files $uri $uri.html =404;
}

Now don't forget to update your host file: sudo nano /etc/hosts -- this will open your hosts file. Add an entry for each dev domain you use:

127.0.0.1    mysite.dev

Ctrl+O, Enter, Ctrl+X to save and exit. Make sure to reload nginx: sudo nginx -s reload. Now when you type http://mysite.dev in your browser, it will talk to your localhost nginx which should serve up that site based on the domain you are requesting.

If your dev site uses PHP...

...then some special stuff needs to be added to your site config file. To have nginx talk to PHP through FastCGI, my conf file (sites-enabled/mysite.dev.conf) looks kind of like this:

server {
  listen 80;
  server_name mysite.dev;
  root /Users/.../no-trailing-slash/directory;
  index index.php index.html;

  location / {
    #try_files $uri $uri/ /index.php;  # this line was causing the index.php file to be loaded twice...
  } 

  location ~ \.php$ {
    fastcgi_split_path_info ^(.+\.php)(.*)$;
    fastcgi_index index.php;
    include fastcgi.conf;
    if (-f $request_filename) {
       fastcgi_pass 127.0.0.1:9000;
    }
  }

  try_files $uri $uri.php $uri.html =404;
}

Again, don't forget to reload nginx when you make changes.

Last thing: we need to tell nginx where to find translation information for FastCGI, so it knows how to talk to it. Run this command: sudo cp /opt/local/etc/nginx/fastcgi.conf.default /opt/local/etc/nginx/fastcgi.conf

Don't forget to reload nginx!

Run php-cgi (FastCGI) when the Mac boots

Almost done! PHP's CGI needs a little help in order to run when the Mac boots up. This gave me a headache for hours after banging my head against the desk. It's actually simple using launchctl with a plist file all ready for you:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.macports.php-cgi</string>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/local/bin/php-cgi</string>
    <string>-b127.0.0.1:9000</string>
    <string>-q</string>
  </array>
  <key>EnvironmentVariables</key>
  <dict>
    <key>PHP_FCGI_CHILDREN</key>
    <string>2</string>
    <key>PHP_FCGI_MAX_REQUESTS</key>
    <string>256</string>
  </dict>
  <key>RunAtLoad</key><true/>
  <key>Debug</key><false/>
  <key>KeepAlive</key><true/>
</dict>
</plist>

I put this in a file: /Library/LaunchDaemons/org.macports.php-cgi.plist. Important: No spaces allowed between "-b" and "127.0.0.1:9000." I don't know why. But that's just how it is. You're welcome to customize anything else if you want.

To finish, then, run: sudo launchctl load org.macports.php-cgi.plist

Now the FastCGI wrapper for PHP will load when your system does, along with MySQL and nginx.

Restart your system and try loading your PHP dev website in your browser! (e.g. http://mysite.dev) -- everything should be working. If not, check your system console to see if FastCGI is failing to start and re-spawning every 10 seconds. That means you have a problem in your plist file.

If MySQL can't connect via PHP and you get a funny error on mysqli_connect kinds of functions, try switching the host from "localhost" to "127.0.0.1" in your connection parameters in your PHP script. This might be related to the plist file we created or something in hosts, maybe even the default MySQL config. I'm not sure, but switching that fixed it for me.

Have fun on your shiny new dev environment!
Read More
Posted in development, mysql, nginx, osx, php, terminal | No comments
Older Posts Home
Subscribe to: Posts (Atom)

Popular Posts

  • How to take FrontRunner from Provo to SLC airport
    I see this question a lot: how do I get from Provo or Orem to the SLC International Airport entirely by train (UTA FrontRunner/Trax)? Here I...
  • Behavior-driven testing in Go with GoConvey (BDD in "golang")
    First: the built-in Go testing tools Few things bring sweeter peace to the soul than making changes to Go code, then: $ go test ... PASS ok ...
  • Fix the Home and End keys on Mac OS X (Mountain Lion)
    If you use a keyboard that's not designed specifically for Macs, you probably are familiar with the annoying mapping of the Home and End...
  • Why yes, Go/Golang, I still want to read my CSV file!
    UPDATE: This appears to have been fixed and the fix  ships with   Go 1.2 . I like Go (1.1.1), but how disappointing that in order to read ...
  • Installing nginx / PHP / MySQL on Mac OS X Mountain Lion
    ** Update: See a quicker way to do this using Homebrew (this method uses Macports, and it's considerably more difficult). ** ... are yo...
  • Using Vagrant and cross-compiling Go (golang)
    This is mostly a memo-to-self about how to write Go code in my Mac environment, compile it there for a Linux environment, and run it in a pr...
  • Automatically make a Raspberry Pi with wifi support
    Okay, I love my Raspberry Pi, but setting it up just the way I want got so involved I was afraid I couldn't do it again if I had to. So ...
  • Install nginx / PHP / MySQL on Mac OS X Mountain Lion with Homebrew
    Last time I wiped my Macbook Pro, I used Macports to install my web development environment . Doing it that way was really hard compared to ...
  • Printing to BYU campus printers without extra software (Mac OS X)
    Macs come with a print server installed by default. This means you don't need to install any drivers or software to print to the BYU ca...
  • External hard drive backups while you sleep
    On most modern computers, there's an energy saver preference which will shut down your hard disks when the computer is idle or in "...

Categories

  • backup
  • bdd
  • byu
  • chrome
  • cli
  • command line
  • commute
  • compile
  • cross-compile
  • csv
  • development
  • dns
  • domain name
  • fastcgi
  • fcgi
  • go
  • golang
  • homebrew
  • inkscape
  • install
  • ip
  • ip address
  • ipaddress
  • itunes
  • javascript
  • keybinding
  • linux
  • mac
  • mountain lion
  • mysql
  • nginx
  • optimization
  • osx
  • parsing
  • pecl
  • php
  • printing
  • raspberry pi
  • security
  • ssd
  • terminal
  • testing
  • transportation
  • unit tests
  • vagrant

Blog Archive

  • ▼  2013 (16)
    • ▼  November (1)
      • How to take FrontRunner from Provo to SLC airport
    • ►  October (1)
    • ►  September (2)
    • ►  August (2)
    • ►  July (3)
    • ►  June (1)
    • ►  May (1)
    • ►  April (2)
    • ►  March (2)
    • ►  February (1)
  • ►  2012 (8)
    • ►  November (1)
    • ►  October (1)
    • ►  September (6)
Powered by Blogger.

About Me

Unknown
View my complete profile