Archive for the ‘google’ Category

Kandy: a Google Reader app for your Kindle

March 14th, 2013 Comments off

For the last couple of weeks I’ve been working on a Google Reader web app for the Kindle. I love my Kindle and I use it for reading wikipedia and blog posts too. Lately I was using it to read Google Reader but the interface, aimed for desktop computers, was hard to use with the Kindle.

So, as a pet project, I started to code a new web app that could access the Unofficial Google Reader API and present the subscriptions and entries in a more pleasant way.

Today Google announced it will discontinue Google Reader on 1st July. Kandy is still very basic but it’s functional. There were things I wanted to polish and I got some other ideas in mind, but with today’s news I don’t feel like investing more time on this app.

You can check Kandy online here: (better access from your Kindle :)). The code is available at github:

Enjoy it while you can :)


Categories: google, igalia, javascript, web Tags:

Reverse engineering Google Reader

October 5th, 2008 No comments

This is an old topic. Google Reader is one of the few Google products which lacks an API. However, some people managed to reverse-engineering it almost three years ago. There is a well-know post (Google Reader API) from Niall Kennedy, former employee at Technorati, where he explained the underpinnings of Google Reader.

Comments on that same post are also interesting. One of them links to the docs of pyrfeed, a RSS/Atom framework which uses Google Reader as storage base. In my opinion, this is the best documentation I found so far about how the protocol works [1].

Interestingly enough, as some googlers admitted, Google Reader was in fact built upon its own API, that means the API came first, however, it has never been released. I believe that since it was one of the first Google products which was bound to an API, there was not a standarized way of doing things at that time, as an outcome, there are a few things which differ from other Google APIs. One of these differences, for example, is how to fetch an user authorization token, which is not based neither on ClientLogin or AuthSub.

Despite the completeness of pyrfeed documentation, I could tell from the comments on Nial Kennedy’s post, that there are still a few developers who do not know how to retrieve an authorization token, and how later, pass this token on requests which may need authorization, like getting a list of your feeds, managing them, or making a feed as starred. Let’s see how it works:

If you wish to perform an operation in Google Reader which requires authentication, you need to provide the following two tokens:

  • SID (Session ID)
  • Authentication token or T token (do not mix up this token with the Auth token you get by authenticating with ClientLogin)

There are also two ways of retrieving a Session ID token :

  • If your browser (i.e Firefox) is opened and already logged in any Google site, a valid SID token would be stored in it. Firefox stores its session tokens inside sessionstore.js file. In conclusion, search for the SID inside sessionstore.js if there is any google session opened inside your Firefox.
  • Authenticating against Google reader service using ClientLogin mechanism. Google reader service codename is reader. ClienLogin mechanism is fully documented by Google. ClientLogin mechanism returns three tokens, one of them is the SID token.

Once you got a valid SID token, do a GET request to the following URL to get a valid T token.

Please notice that you should add the SID token as a cookie on this request, otherwise it would fail.

Once you get both tokens, SID and T, you can successfully perform any Google Reader operation which requires authentication (always pass these two values as cookies)

Finally, let’s review the whole proccess by examples:

1. First, get a valid SID token:

-d Email=just_your_username_here_without_at_gmail_dot_com
-d Passwd=your_password_here
-d source=Google-cURL-Example
-d service=reader


2. Once you get a valid SID, request a T token:curl -s -X GET –header “Cookie: SID=DQAAAH0AAAC0YHom0L5LDq10xGnbQK_O7OLiX3Qrou4XeA6P469shoM1goEFQT_zVn8YxDV38Y5v3mGJlhSzJuz5xLPqpKEM0Wedks-ak7LLpNjO7dZw779ljOQrC-2UCYFjiktJcfXmof7WeZs7O0SCNCQgPSKaENJ6FBTeDBeQLahUUrajrg”


3. Once you get a valid T token, perform any Google Reader operation you want, for instance, retrieve a list of your feeds

curl -s -X GET –header “Cookie: SID=DQAAAH0AAAC0YHom0L5LDq10xGnbQK_O7OLiX3Qrou4XeA6P469shoM1goEFQT_zVn8YxDV38Y5v3mGJlhSzJuz5xLPqpKEM0Wedks-ak7LLpNjO7dZw779ljOQrC-2UCYFjiktJcfXmof7WeZs7O0SCNCQgPSKaENJ6FBTeDBeQLahUUrajrg; T=kArWxxwBAAA.vegUbtUjv2Vvf_HKWlwjIA.QfhP1LoSb5ghYPz_AbOG_Q” | tidy -xml -indent -quiet


[1] pyrfeed, Google Reader API

Categories: google Tags:

Uploading Google Internet Gadgets from command-line

September 28th, 2008 No comments

If you had ever been curious about how to code a Google Internet Gadget, you most likely had put your hands on programming tools such as Google Gadget Editor for iGoogle.

I had coded a few gadgets so far. Although I didn’t spent much time with the editor, there were a couple of, let’s say features, I disliked getting the feeling that they were slowering down my work letting me be, in turn, less productive. Things like:

  • Automatic refreshing, opened helloworld.xml sample to start coding my gadget, and stayed idle for a couple of minutes you get helloworld gadget refreshed throwing all your effort till then right to the dustbin.
  • Cursor is set to first position of the text editor box after saving a file, super-inconvenient.
  • On my Lenovo T61 laptop, pressing the key which is just left to the upper-arrow key, made the gadget to reload without confirmation. I know, this is more related to my own my settings, but cause me much headache (since it’s easy to rubber the key by chance when moving upwards)
  • All the inconvenience of using any other text-editor when you are a Vim lover.

All this led me to code a command-line Perl script to upload a Google Internet Gadget to iGoogle.

There is not such a thing as an API for iGoogle to help programmers to interact with iGoogle, so all work has to be coded in an old-school fashion, pretending your script is in fact a browser and has to communicate with the iGoogle server in the same manner.

If you are a Perl programmer, you most likely be familiar with popular packages such as LWP (lib www Perl, although this acronym always reminds me to Larry Wall’s Perl). Client-Server HTTP communication defines a client as an user-agent, and that is all what LWP is about. First, instance a class called UserAgent, think of it as a XMLHttpRequest object is you are familiar with Javascript, then codify your http request, send and wait for a response.

Before stating coding, I used Wireshark, formerly known as Ethereal, to peer at how Firefox should interact with iGoogle. After that, the goal is to gathered all needed information iGoogle would expect from a browser request. The most difficult part here was to figure out were to get the SID cookie from.

Many of the cookies iGoogle expect from an HTTP request can be retrieved from your cookies.txt file (or querying moz_cookies table if you are using Firefox 3). All values stored in cookies.txt are permanent cookies. The SID cookie is in fact a session cookie, and by hence, should be stored in sessionstore.js file. You can also retrieve this value by successfully authenticating against Google using ClientLogin mechanism.

Summarizing, if your browser has an already opened Google Session, our script should peer at sessionstore.js file to fetch SID from it. If there is not a Google session opened within your browser, then our script should authenticate against iGoogle to get that specific value. Google ClientLogin mechanism is aimed just to retrieve a valid Auth token, necessary to interact with all Google’s services and APIs. The docs even claim that “SID and LSID are not currenly activated ans should not be used“, but here you see how this SID token can be used to successfully interact with Google’s services (upload files to iGoogle, Google Page Creator, etc) which lack an API.

Script to upload Google Internet Gadgets to iGoogle: download.
You need to set the following variables:

  • username, you Google user name (without
  • password, your Google account password

NOTE: If a Google session is opened within your Firefox browser (that means, you are already logged in at least one Google service, when you use this script), then you can leave this two values empty.

  • GG_ID (Google Gadget ID), every iGoogle user has a Google Gadget ID. To know your GG_ID, you must figure out what your target URL looks like. When the Save… option is pressed, Google Gadget Editor sent a POST request to a target URL whick looks like:$GG_ID/$filename?synd=gde&et=itJbVyUh


  • filename, is the name of file to upload
  • GG_ID, is your Google Gadget ID.

Use Wireshark or Firebug to figure out what is the value for your GG_ID. Please, let me know if there is a better way to get this value.

Perl libraries required:

  • DBI
  • Data::Dumper
  • Config::General
  • LWP

NOTE: I tested this script on Firefox 3 (remember is needed to fetch cookies from whatever cookie store mechanism, sqlite3 in case of Firefox3). I added code to read cookies from cookies.txt (cookie store mechanism in Firefox 1 & 2), but haven’t tried it.

Categories: gadgets, google Tags:

Cookies in Firefox 3

August 23rd, 2008 No comments

Cookies are small pieces of data send by an HTTP server to an user agent (generally a browser) and sent back from the client to the server. In general, cookies are used for authentication, user tracking, maintaining user preferences, shopping carts. etc.

Cookies consist of a name/value pair set by a server by adding an extra header to an HTTP frame:

/1.1 200 OK
Type: text/html; charset=ISO88591
: blah:foo
Cookie: PREF=bogus;

Consider a successful response received by a server. The domain asks the user browser to set cookie PREF on save it for later use. All cookies previously set by a server, are sent back on subsequent requests.

Cookies were firstly thought as a mechanism for implementing persistence over HTTP communications. Since HTTP is a stateless protocol, there is no way to keep track of user interactions between different requests. HTTP protocol simply allows a browser to request a single document from a web server, without remembering who this client was. Cookies can be used to identify previous client connections, serving as the basics for implementing HTTP sessions.

There are two types of cookies:

  • Session cookies, a temporary cookie stored in the user agent memory.
  • Permanent cookies, used to store user’s preferences. Permanent cookies are stored in files and last for more than a session.

In Firefox, cookies are stored in ~/.mozilla/firefox/???.default/cookies.txt. Different browsers set a different limits for the amount of cookies a host can store. Firefox lets a domain to store up to 50 cookies per user, each up to 3-4Kbytes. You can view your cookies in Firefox by clicking on Edit->Preferences->Privacy->Show Cookies…

Netscape, and later Firefox till version 2.0, store permanent cookies in a file named cookies.txt. This file has the following file format: TRUE / FALSE 946684799 NETSCAPE_ID 100103

Each line represents a cookie, composed by different fields separated by a TAB. From left-to-right this is what each field represents:

  • domain, The domain that created AND that can read the variable
  • flag, A TRUE/FALSE value indicating if all machines within a given domain can access the variable. This value is set automatically by the browser, depending on the value you set for domain
  • path, the path within the domain the variable is valid for
  • secure, A TRUE/FALSE boolean value indicating an https connection with the domain is valid to access the variable
  • expiration, The UNIX time that the variable will expire on. UNIX time is defined as the number of seconds since Jan 1, 1970 00:00:00 GMT
  • name, name of the variable
  • value, value of the the variable

Imagine for a moment you log in Gmail by accessing to From there you are redirected to, which set the following cookie in your browser. TRUE /gmail TRUE 0 SID DXV9AAAA0G…

URL was split in two parts: domain and path, setting host to and path to /gmail, respectively. SID is in this case the name of this cookie, and very long string, DXV9AAAA0G…, its value. What’s more, flag is set to TRUE which means this cookie, SID, can be shared between different applications within the domain. The highest value a UNIX timestamp can represent is sometime in year 2038. Originally, Google cookies were set to this value, though to last more than 30 years from now. Privacy advocators complains have recently led Google to dramatically reduce theirs cookies’ lifespan to just 2 years [1] .Lastly, secure field is set to TRUE which means an https connection is required.

Firefox 3 features a SQLite3 database engine for storing permanent data such as user preferences, bookmarks, cookies, and many more. More specifically, cookies are stored in {profile_folder}/cookies.sqlite.

A quick glimpse to this database reveals a single table, named moz_cookies, with the following data structure:

CREATE TABLE moz_cookies (id INTEGER PRIMARY KEY, name TEXT, value TEXT, host TEXT, path TEXT,expiry INTEGER, lastAccessed INTEGER, isSecure INTEGER, isHttpOnly INTEGER);

Since cookies are now stored in a database, field order no longer is relevant.

  • isHttpOnly, true if the cookie should only be sent to, and can only be modified by, an http connection.
  • lastAccessed, is a new field indicating when was the last time this variable was accessed.

Conversely, session cookies are saved in a different file. Both Firefox 2 and Firefox 3 come with a built-in Session Store feature that saves your session data including open windows and tabs, window size and position, and text typed in forms. Session data is stored in the sessionstore.js file, located in your profile folder. So, in the example given above, SID (session id) cookie will be stored in sessionstore.js, rather than in cookies.txt or cookies.sqlite database.

In general, every web server sets a SID cookie whenever an user logs in successfully. Google, for instance, set a SID cookie in the user’s browser once he or she logs in, or programmatically by using some of Google’s authentication mechanism: AuthSub or ClientLogin (check my previous post: “Authenticating against Google services” to see how a successful ClientLogin response looks like.) Now it seems not difficult to figure out what that SID cookie actually meant, right?

[1] The official Google Blog, Cookies: Expiring sooner to improve privacy, 2007
Marty Stepp, Cookies and Sessions, Web programming, University of Washington, 2007
David Whalen, The Unofficial Cookie Guide

Categories: firefox, google, web Tags:

Authenticating against Google services

August 16th, 2008 No comments

Google provides two ways for authenticating third-party applications against Google services:

  • AuthSub
  • ClientLogin

AuthSub is aimed to third-party web applications that may need to access an user Google service, such as Google Calendar, Picasa Web Albums, or other Google services. AuthSub authenticates an user via a proxy web page. Only when the service is about to use, the user is redirected to a Google’s authentication page for typing their credentials. Once the user has successfully logged in, the browser is redirected back to our web application.

On the other hand, ClientLogin was thought as an authenticating mechanism for standalone desktop applications, where redirecting to a web page may not be possible or does not make much sense. Users must rely on third-party applications to handle their Google user-name and password safely.

Let’s leave AuthSub aside and take a look at ClientLogin and how it works. First of all, before interacting with a Google service it is necessary to be authenticated against that precise service. ClientLogin is the mechanism we are going to use to do it so. It needs at least four parameters:

  • Username, your Google’s account user-name (without
  • Passwd, your Google’s account password.
  • service, the name of the Google service you want to use.
  • source, and identifier for an application, composed of the following values: “PluginApplication nameVersion“;

Let’s compose an HTTP frame for authenticating against Picasa Web Albums

-d Email=john.edwards
-d Passwd=newfoundland
-d source=Google-cURL-Example
-d service=lh2

If the authenticating process is success, that means, we got a 2XX status code back, the result would look something like this:


As an outcome, we got three cookies or tokens. Among these three, the cookie labeled Auth represents the authentication token we are interested in, and should be used here after in every further interaction with the service. SID and LSID cookies are not currently active and should not be used (which is partly true, more on this in a coming post).

Several things to highlight before continuing:

  • In general, while interacting with web services or composing hand-made HTTP frames is always a good idea to turn WireShark on for debugging.
  • Notice that every Google service has its own identifier. In the example above, lh2″ identifier stands for Picasa Web Albums service. A more detailed list of Google services and its code names can be found on this post: “Google Account Services Names”. Whenever we need to interact with a Google service is necessary to authenticate against that Google service. Right now, there is not such thing as single-sing-on mechanism for interacting with all Google services.
  • Parameters are case-sensitive, both Email and Password start by a capital letter, whereas source and service are lowercase.
  • About the -d modifier, indicates the name and value of an <input> element in a HTML form. If you want to learn more about how to compose HTTP frames with cURL check the following article: “The Art of HTTP Scripting”.

Once we got a valid Auth token, we can start using the service. All Google services work in a similar way. In general, we can perform any CRUD (Create-Read-Update-Delete) operation over them, in a Restful style, thanks to the most common HTTP methods: GET (for reading), POST (for creating), PUT (for updating) and DELETE (for erasing).

Let’s request a list of albums from my Picasa Web Albums account.

curl --silent -X GET | tidy -xml -indent -quiet

Usually for reading operations there is no need to authenticate in before-hand neither providing any Auth token in the HTTP frame.

On the contrary, managing operations such as creating new data, deleting or updating, need an Auth token. Let's compose an HTTP frame using the Auth token retrieved in  the first step. This operation will create a new album, together with its description, in my Picasa account.

curl --silent -X POST --data "@album_entry.xml" --header "Content-Type: application/atom+xml" --header "Authorization: GoogleLogin auth=DQAAAIAAAACQCCOvLjNU4y6zYclO4t75sjAWtWxx3C6OdYbMz3WXjn6rux_Jb3O25xzy5RCktrwH4oB2rKe3SsPq81ee2Pnv2VkPO_XrKYeuIP4dIqyuCsrlg1c4bZ4f5zTXD6GcMSTYfpts8KwR4m4Bn1x5siTq7ERpgXxqrOvZameELOhFEw" "" | tidy -xml -indent -quiet

where album_entry.xml is a file containing an album description in GData file format (as for the XML format of entries, GData is strongly based in Atom plus a customized set of tags added by Google). Check the Picasa Web Album Data API documentation for further information. 

<entry xmlns=''
<title type='text'>A day at the opera</title>
<summary type='text'>Funny pictures taking on the Faemino and Cansado show last December.</summary>
<media:keywords>theater, show, humour</media:keywords>
<category scheme=''

Lastly, let’s delete the album entry we just created.

curl -s -X DELETE --header "Authorization: GoogleLogin auth=DQAAAIAAAACQCCOvLjNU4y6zYclO4t75sjAWtWxx3C6OdYbMz3WXjn6rux_Jb3O25xzy5RCktrwH4oB2rKe3SsPq81ee2Pnv2VkPO_XrKYeuIP4dIqyuCsrlg1c4bZ4f5zTXD6GcMSTYfpts8KwR4m4Bn1x5siTq7ERpgXxqrOvZameELOhFEw" ""

where 5234863202168634657 stands for the albumid.

In a nutshell, Google provides a Data API for many of their services. It is possible to use most Google services in a programmatic way. Before using any service, it is necessary to authenticate against it. If we were success, this authentication process will provide an Auth token as a result. This token would be needed later on for interacting with the Google service we just authenticate against. It is important to notice, there are two valid methods for retrieving an Auth token: AuthSub and ClientLogin. As a rule of thumb, use AuthSub for web applications, and ClientLogin for standalone applications. Remember, whenever interacting with a service, especially when creating new data, modifying it, etc it is necessary to embed a valid Auth token as a header in the HTTP frame. The Auth token must be present for every interaction. Auth tokens cannot be shared among different services, every service requires its own valid Auth token.

The AuthSub authenticating process is fully described in the following link: AuthSub Authentication fow Web Applications” , whereas  more detailed information about ClientLogin can be found here: “Authentication for Installed Applications”.

Categories: google, igalia, web Tags:

Google gadgets and OpenSocial

August 7th, 2008 No comments

Gadgets are small applications, general with an UI, which perform a very specific task. In the Java world, for example, they are known as applets, and were very popular when Java started. However, due to poor Java performance, applets never become very successful and eventually failed. The attempt of bringing RIA applications to the browser world was postponed till Ajax came into play.

Google gadgets is Google approach to gadgets, and it comes in two flavors:

  • Google Desktop Gadgets. Features a google-like search engine for the desktop, and a rather wide toolbar or panel where to stick small applications, that means, gadgets. There is a wide variety of gadgets ranging from TODO lists to weather forecasts and music control players. Honestly I think is nothing new, we got them in Linux since WindowMaker or Afterstep, but what make gadgets special are that they are thought to be easily integrated with internet services.
  • Google Internet Gadgets, which are those we can add from iGoogle. iGoogle aims to be a perfectly customized front-page for websurfers, where they can organize and arranged gadgets providing the services they need. It is possible to add new tabs, group gadgets by category, and access to tons of information without bothering getting to a specific web page. Although the potential is huge, most gadgets are just RSS/Atom feeders.

It is also important to highlight that Google is not the only provider offering this kind of tools, other sites such as NetVibes are also very popular among users.

Both type of gadgets are based on Javascript/HTML plus and specific XML language from Google for managing settings. In the case of internet gadgets, an API for easing Javascript programming, or fetching contents from the web is provided. The desktop gadgets SDK, comes as well with an API but more oriented to bring mechanisms for communicating gadgets with desktop resources such as video, audio, system, etc. Standard Javascript programming is recommended for building desktop gadgets (by using standard Javascript we can guarantee our gadget will run in different environments). By the way, Google desktop gadgets have been released recently for Linux under an Apache Open-Source License.

These last days, Google internet gadgets have brought their attention back to me. Watching David Glazer’s talk at Google IO: “OpenSocial Specification: What’s Next for OpenSocial”, I learnt google internet gadgets will become one of the pillars OpenSocial will rely on. OpenSocial is an initiative based on open standards, with the goal of providing a non-proprietary technology framework where to build social-enabled applications. Basically, is the attempt of Google plus other social-network sites, some of them big players in the industry such as Linked-in, MySpace, Friendster, SalesForce, etc…to fight against the fearing and growing dominance of Facebook as social platform.

From a technological point of view, Facebook gadgets are server-oriented. That means, they must be hosted in a server, where they work alike any other web application. Since Facebook is mostly programmed in PHP, Facebook gadgets are generally programmed in PHP as well, although bindings for other languages exist. Despite the need of a server where to host our Facebook mini-application, gadgets have not stopped springing out since the Facebook Developer Platform was released. The Facebook developer community looks very active with lots of events being organized worldwide (take a look at Facebook Developer Garage Barcelona). Talking about Facebook apps, I found very revealing this article from Ben Lorica, “Facebook App Categories Ranked By Usage”, at O’Reilly Radar. If you are interested in social networks I strongly encourage you to take a look at this figures (I would like to see other numbers from other social sites).

On the contrary to Facebook gadgets, Google internet gadgets can run either on a client as well as in a server. Of course, they are hosted in a server, more precisely in some hard-disk at Google’s facilities, but you do not need of an external provider where to host your gadget, as it happens with Facebook. You can do all your work by maybe interacting with other internet services and mixing contents together. In case you need to store user specific data, then you may need a server (basically a storage provider and an API to interact with).

Finally, although it lacks social-enabled features I tried Google internet gadget API and this is the result: Go Movies! It is a gadget for checking what is on cinemas nearby your town. It retrieves data from Google movies, and present them as a list of movies (can pick movie theater), adding new links to IMDB for movie information and YouTube for trailers. Unfortunately, it seems Google movies is one of those obscure Google services (like Google music search) which have not been yet gdatized. To turn things worse, the html result page is not well-formed, that means any chance of querying the document using XPath should be discarded. I went old-school for this time and parse the document using regular expressions (luckily enough, Javascript regexp are 99% similar to Perl regexp). Some parsing fails when a ñ or Ñ character occurs or when UTF-8 characters happen (sorry for my beloved Japanese xDD).

Further reading:

At the Forge – OpenSocial and Google Gadgets

Categories: google, igalia, social, web Tags: