Damon Clinkscales' Stumblings

I have been playing with CouchDB for the last couple of days a lot. And I like what I see even more. Couch seems to be extracted from the web so much, that it reminds me of my joy when discovering Rails couple of years ago.

One of those impossibly great things is that CouchDB is completely „raw HTTP“-based.

So, for instance, clustering is one of those things achieved so trivially it almost hurts — because it’s just the same thing we are used to do by fronting Mongrel instances with Apache/Nginx, etc.

Let’s try something out:

# -- Configure cluster A
$ cat /opt/local/etc/couchdb/serverA.ini
[couchdb]
database_dir = /tmp/serverA
[httpd]
port = 6001

# -- Configure cluster B
$ cat /opt/local/etc/couchdb/serverB.ini
[couchdb]
database_dir = /tmp/serverB
[httpd]
port = 6002

# -- Start both clusters
$ sudo couchdb -a /opt/local/etc/couchdb/serverA.ini
# [info] [] Apache CouchDB has started on http://127.0.0.1:6001/
$ sudo couchdb -a /opt/local/etc/couchdb/serverB.ini
# [info] [] Apache CouchDB has started on http://127.0.0.1:6002/ #

# -- Configure Nginx as reverse proxy
$ cat /opt/nginx/conf/nginx.conf
http {
  # ...
  upstream couchdb_cluster {
    server 127.0.0.1:6001;
    server 127.0.0.1:6002;
  }

  server {
    listen 80;
    server_name couchdb_cluster.local;

    location / {
      proxy_pass http://couchdb_cluster;
      break;
    }

  }
  # ...
}

# -- Map the hostname in /etc/hosts
$ cat /etc/hosts
# ...
127.0.0.1 couchdb_cluster.local

# -- And start Nginx
$ sudo /opt/nginx/sbin/nginx

OK. That’s the whole cluster setup. Lets create some database on both nodes:

$ curl -X PUT http://localhost:6001/my_database
$ curl -X PUT http://localhost:6002/my_database

Now we have identically named databases on both instances. Time to set up some continuous replication between them?

$ curl -X POST http://localhost:6001/_replicate \
    -d '{"source":"my_database", "target":"http://localhost:6002/my_database", "continuous":true}'
$ curl -X POST http://localhost:6002/_replicate \
    -d '{"source":"my_database", "target":"http://localhost:6001/my_database", "continuous":true}'

Just be aware, that according to the CouchDB book, continuous replication currently does not survive server restart. Future versions of CouchDB may allow you to create target database while setting up the replication, according to Jan Lehnardt.

Now open the Futon graphical interface for the databases on http://localhost:6001/_utils/database.html?my_database and http://localhost:6002/_utils/database.html?my_database. You should see an empty database in both.

Let’s create some document in one them, let’s say A, either by curl or in Futon:

$ curl -X POST http://localhost:6001/my_database/ -d '{"foo":"bar"}'

When you refresh the browser, you’ll see the document in both of them. Now, pick one of the cluster nodes, and change the document in Futon. You’ll see the change synchronized to the other one instantly.

Magic? No. „Any sufficiently advanced technology is indistinguishable from magic.“

You can now point your client application (or many clients) to the couchdb_cluster.local frontend, and Nginx would do simple load balancing between the nodes. Now, something like HAproxy or reverse-proxy cache would put the thing on completely another level.

This is just an example how the fact that CouchDB is HTTP-based enables you to re-use your knowledge from one area („how scaling of Rails apps is done“) to another.

CouchDB just makes sense.

Earlier today, we accidentally suspended a number of accounts.

We regret the human error that led to these mistaken suspensions and we are working to restore the affected accounts—we expect this to be completed in the next several hours.

One additional note: some the accounts suspended were using the third-party site Tweetlater. However, Tweetlater is not to blame for these suspensions nor is it in violation of our Terms.

We’re working to reduce the amount of time it takes for new follower/followings to take effect. Right now, after you follow or unfollow someone it may take some time for that action to be properly recorded and propagated throughout the system. As a result, you may run into discrepancies such as:

• follower and following counts are off
• new followings not showing up in your timeline
• seeing that you can follow someone you’ve already followed

We’re planning several upgrades to address these issues and will post more as we make progress.

Update (3:03p): Other inconsistencies in addition to the above also exist:

• device notification changes for followers (choosing whose updates you get on your phone)
• favoriting and unfavoriting of individual tweets