Thursday, 6 October 2011

Time-zone database down

Today (2011-10-06), the time-zone database was closed down.

It is perhaps easy to read that line, think it doesn't affect you, and then move on. But thats just not the case.

Update 2011-10-07: A backup copy of the mailing list and subscribers has now swung into operation run by volunteer Robert Elz. Please only join this list to discuss changes to time-zones - its not a "fighting the lawsuit" list. New tz list

Update 2011-10-07: More details on the history of ACS, Astrolabe and how this came about from a competitor. Worth reading.

Update 2011-10-10: A new version of the time-zone database 2011l has been released. This news indicates how other members of the community have stepped up and continued the work, so the database can no longer be considered to be "down". However, we all need to ensure that we continue to support Olson and Eggert in their fight with the foolish people at Astrolabe. My blog about the rebooted database.

Update 2011-10-26: The situation has now stabilised. The database is now hosted by IANA after the succesful database reboot. Astrolabe also gave their opinion on the matter.

The time-zone database (sometimes referred to as the Olson database) is the computing world's principle source of time-zone data. It is embedded in every Unix and Java for starters, and will be used by many websites and probably by your iPhone. You may know it via the IDs, such as "Europe/London" or "America/New_York".

But, perhaps you're thinking that time-zones don't change? Well that may be true for America and the EU right now, but certainly isn't for the rest of the world. Governments change their time-zones all the time, and the decisions are frequently very political. I'd estimate there are between 20 and 100 separate changes made around the globe each year. And these can be at very short notice, triggered by earthquakes for example.

The time-zone database tracks all this information and creates a standard format file that describes it. I would show you an example of the file, but then perhaps I'd be sued....

The database itself was run as an open source project, led by Arthur David Olson, supported by many others. The data was published as a set of files about 15 times a year, and then picked up by users everywhere.

The complaint itself comes from Astrolabe, Inc, whose website looks like a company I would avoid doing business with.

The complaint is that Astrolabe produce a work, the "ACS Atlas", which is referenced by the time-zone database (some sources suggest that Astrolabe may have recently purchased the work). Astrolabe claim copyright over their work and thus believe that the time-zone databse should not have released their information to the public domain. The case is targetted at two private individuals - Arthur David Olson and Paul Eggert, who have hosted the website for many years.

The key passage in the time-zone database files is this:

# From Paul Eggert (2006-03-22):
# A good source for time zone historical data in the US is
# Thomas G. Shanks, The American Atlas (5th edition),
# San Diego: ACS Publications, Inc. (1991).
# Make sure you have the errata sheet; the book is somewhat useless without it.
# It is the source for most of the pre-1991 US entries below.

For obvious reasons, I'll refrain on commenting on the rights and wrongs of the case, although I will note that facts like the phonebook cannot be copyrighted. A detailed response from one site taken down is now available. Instead I'll focus on the impact.

The impact of this is severe for anyone that uses it - whether via Java, Unix or some other means. This really is the key tool used by everyone to tell the right time globally. We all owe a debt of gratitude to the database maintainers who have worked on this for many, many years at zero cost to the industry and for zero financial gain.

So, right now the global situation is that there is no longer a single central location for time-zone information for computing. I'm sure that each major user project (like the Unix distros) will patch their own versions as best they can, but the stricter ones might argue that the current data is tainted and want to remove even that. This could get very messy very quickly.

Both Joda-Time and ThreeTen/JSR-310 use the data to build timezone information. ThreeTen/JSR-310 in particular provides this information in huge detail to applications. The worst case scenario is that multiple groups start up to provide this data in the future, and applications are then responsible for handling multiple competing data sources.

This data is so key to the world at this point that it needs to be formalised and run by a group with more legal and financial backing. Efforts had been ongoing to achieve this, but they may now be in jeopardy - who would want to take on a project being legally attacked?.

I hereby call on the industry leaders to help sort this out - IBM, Oracle, Apple, Google, RedHat I'm looking at you.
Update: I didn't include Microsoft here because Windows has its own time-zone data files.

In the meantime, could I please ask that anyone thinking of patching the data on a temporary basis, or trying to recreate it from scratch, re-uses the existing file format. There is no reason to believe that the C code or file format is tainted by the lawsuit, just the data. So, lets all please try to minimise the mess that could happen if everyone starts to go their own way.

Add a comment or a rant below!

Update 2011-10-07: Alternatively you can rant on their Facebook page, Website contact us page or Amazon book review (note that the book may be by a different company - its a little unclear).

67 comments:

  1. Hi Stephen - I'll have a chat to the JCP EC about this - we're not the right body to host the data, but the EC contains the range of big industry leaders that can form something to host this. Drop me a line if I don't get back to you within the week about this. - M

    ReplyDelete
  2. [quote]
    But, perhaps you're thinking that time-zones don't change? Well that may be true for America...
    [/quote]

    You, sir, have obviously not heard of Indiana.
    https://secure.wikimedia.org/wikipedia/en/wiki/Time_in_Indiana

    Indiana time zones were even featured as a plot point in an episode of The West Wing.

    ReplyDelete
  3. I wouldn't be surprised if the computers of Astrolabe needs it too. Speaking of irony...

    ReplyDelete
  4. David Braverman mention that he "...will be contributing to Olson's and Eggert's legal defense fund once they get it set up." If you happen to find out the details for contributions to the fund, could you please post them.

    ReplyDelete
  5. Databases are copyrightable as they should be.
    http://www.bitlaw.com/copyright/database.html

    ReplyDelete
  6. Okay, so what we need is a website where citizens of different countries can report in the time zone that their country is using, and compile that information into a useful format.

    @carterson2: Did you read all of that?
    http://www.bitlaw.com/copyright/database.html#data
    ^ Directly contradicts what I assume your point was, especially in this case.

    ReplyDelete
  7. > I hereby call on the industry leaders to help sort this out - IBM, Oracle, Apple, Google, RedHat I'm looking at you.

    Isn't it weird that Microsoft doesn't make it onto a list of industry leaders anymore?

    ReplyDelete
  8. carterson2, this wasn't a database that was copied (they even wrote that the compilation alone is useless - you needed the errata sheet). From your link:

    The Copyright Act specifically states that the copyright in a compilation extends only to the compilation itself, and not to the underlying materials or data. 17 U.S.C. § 103(b). As a result, compilation copyrights cannot be used to extend copyright protection to ideas or facts that are otherwise unprotectable (it is a basic premise of copyright law that there is no copyright protection for ideas and basic facts

    ReplyDelete
  9. > Isn't it weird that Microsoft doesn't make it onto a list of industry leaders anymore?

    I assume Microsoft isn't as affected by a *Nix timezone database

    ReplyDelete
  10. It's not that Microsoft isnt an industry leader per se. They are just a notable exception in that they have compiled their own proprietary time zone database and don't rely on the tz data base that is the subject of this post.

    ReplyDelete
  11. .. who knows: maybe microsoft simply has a program to convert that unix tz database into the internal propriatary microsoft format. Would save them quite a bit of work to simply do it that way round. And still maintain their incompatibility with *nix.

    ReplyDelete
  12. NicholBrummer: Nice plot idea actually. Even if MS is evil guess they have other work to do than just work on their "incompatibility" ;) But nice idea, really...

    ReplyDelete
  13. Seriously. Not the end of the world. Someone will simply work around it or create an open database. The judge will probably give them a slap on the wrist for the infringement and the world keeps turning. I think OSX, Linux, Solaris, etc will be just fine. Sometimes I think the open community automatically assumes a company is out to get the little guy just because. Has anybody considered the fact that maybe they DID infringe on something owned by the company? I doubt it is the evil map making giant trying to destroy the world with miscalculated time in *nix Operating Systems. If they own it... then it's theirs. Make your own.

    ReplyDelete
  14. Wait... am I missing something here.. can't they just remove the pre-1991 entries from the file?

    ReplyDelete
  15. Actually it seems the published of the quoted work is http://acspublications.com/ and nothing there suggests that Astrolabe has anything to do with it.

    ISBN of the quoted work is 0935127380.

    ReplyDelete
  16. no you can't remove the pre-1991 entries because you may want to do date calculations on a time-series of data from 1988. for example.

    ReplyDelete
  17. Remove the pre-1991 entries and then re-add them from another source.

    ReplyDelete
  18. Why do I get the feeling they're confusing the book with their own product?

    http://alabe.com/atlas.htm

    ReplyDelete
  19. I've updated the post to explain the absence of Microsoft from the list of industry leaders (they have their own TZ database).

    A number of people have opined various legal theories. I'm staying clear of that as I'm not a lawyer and I don't live in the US.

    While logically we might assume that removing data before 1991 would solve the problem, in all likelihood it wouldn't immediately stop the lawsuit. Whether that is sufficient is open to interpretation.

    A key goal of mine is simply to ask all participants to try to work together on this, and not simply to create lots of forks. If a new database has to be created, lets figure out a way to do it together, perhaps at CLDR.

    ReplyDelete
  20. Have you got in contact with the EFF? I expect they’d be very interested in this case.

    ReplyDelete
  21. There are two products here. The book (ACS publications) and the computer program of the book (Astrolabe). I've checked the address of the Astrolabe website and complaint, and it is definitely Astrolabe (computer program producer) making the complaint.

    ReplyDelete
  22. Part of why microsoft's tz database is often inaccurate in east asian countries. They don't update it nearly as often as the linux one was.

    ReplyDelete
  23. Microsoft don't seem to *have* a correct TZ database. At the moment, Outlook is sending out meeting invitations in US Eastern Daylight Time, which say 'GMT-0500' when in fact EDT is GMT-0400.

    This problem is well-known; they even hacked Exchange to add a disclaimer to meeting invites that it sends out, stating explicitly that the timezone information may be incorrect. One of the most unprofessional and incompetent 'solutions' to a problem that I've ever seen :)

    ReplyDelete
  24. This has got to be the most selfish and stupid thing I have ever seen done on behalf of copyright. What's next? Take down all NTP servers of the world due to supposed "patent infringement"?

    ReplyDelete
  25. I understand that the lawsuit is probably ill conceived, however why was the project completely removed from the web? In a case this egregious shouldn't the project managers seek out a bit of help before shutting everything down?

    ReplyDelete
  26. @caterson2

    Read the document you are linking to:

    "the U.S. Supreme Court ruled that a compilation work such as a database must contain a minimum level of creativity in order to be protectable under the Copyright Act."

    ReplyDelete
  27. I think taking it down from the web is a good. It makes visible how the recent copyright and patent trolling (especially in the US) is destroying internet freedom. And hey, why innovate if you can do rent-seeking on ages-old stuff?

    ReplyDelete
  28. Phone books are actually copyrightable - the company that produces white and yellow pages in Australia took action against another who was straight out copying the information, they won.

    ReplyDelete
  29. "Phone books are actually copyrightable - the company that produces white and yellow pages in Australia took action against another who was straight out copying the information, they won. "

    Sensis recently lost a case against this for the same reason. They failed to prove any creativity went into the making of the database, therefore, it was not copyrightable.

    See: http://www.smartcompany.com.au/intellectual-property/20100212-sensis-loses-case-to-protect-copyright-of-yellow-pages-and-white-pages.html

    ReplyDelete
  30. Update added about Robert Elz's continuation of the time-zone DB work via a new mailing list. (That list is not intended for fighting the lawsuit discussions.)

    ReplyDelete
  31. Should put a rel="nofollow" on your link to astrolabe. Otherwise you're giving them google juice they don't deserve.

    ReplyDelete
  32. Holy... I used to live on cape cod and recognize some folks in their site picture. What a walking talking jokes. I wonder what do they perceive they'll get from the whole thing except from hate?

    ReplyDelete
  33. I've added the rel="nofollow", thanks for the suggestion.

    ReplyDelete
  34. Astrolabe's future prediction technology must have told them they will win the case or they would not have spent money on the lawyer. If the court finds that the ACS is a work derived from public legislation on timezones and they lose, they will have falsified astrology. I for one welcome their application of the scientific method and eagerly await the result.

    ReplyDelete
  35. http://www.astrology-x-files.com/terranatlas.html has a nice write-up on some of the history relating to his, and would seem to support the idea that while some data may have been checked against the ACS data, what has been 'copied' has also been corrected? Perhaps a 'bugs list' with the tz corrected data would be a useful document for the court case? The more evidence that the tz database IS the more accurate work would mitigate against the problem of a straight copy?

    ReplyDelete
  36. OK - just spotted the update that you have found that one :)

    ReplyDelete
  37. I want that company to just die. TZ Data is the more correct and reliable source, they should be proud to be part of this by being quoted. Isn't it prior art to just collect this data and print a book?

    ReplyDelete
  38. WELCOME TO THE NEW AMERICA ;) http://i.imgur.com/Myff0.jpg & http://i.imgur.com/wtjET.jpg

    ReplyDelete
  39. I think that the most interesting point is that the work these people at Astrolabe, Inc. have done has rested on the backs of work shared freely between Astronomers and Astrologers for centuries. I suppose that this is the difference between the people who start the company (with the best of intentions) and the business people who invest and direct the affairs.

    ReplyDelete
  40. It's not like that Astrolabe invented all those timezones, they just talked about it in the past and now nobody else may do so? Now that would be funny.

    If the content of something that profane can be copyrighted I want a copyright on the information that a day has 24 hours. That would just be fair, after all.

    Keep the work up guys and let us know where we can donate to, I'm pretty sure that you won't be left alone. After all this is very important for all the companies out there, too. Imagine that - as a result of this case - you would not be able anymore to talk about history, after all it was written in a book before and as such there must be a copyright on it.

    ReplyDelete
  41. The real shame to me isn't that the TZ database is down (if nothing else, we programmers will find a replacement) - but that Olson and Eggert are being sued for all their great work. Especially with Olson about to step down - we can't let them treat people who spent so long on an open source project badly.

    ReplyDelete
  42. http://en.wikipedia.org/wiki/Trivial_Pursuit#Fred_Worth_lawsuit

    The writer of a trivia book suit the makers of Trivial Pursuit for lifting questions out of his book. He lost because the courts ruled that facts are not protected by copyright. A compilation of trivia questions has an unarguably higher level of creativity than a compilation of historical time zone data, and it still didn't meet copyright criteria.

    ReplyDelete
  43. Nuke (Astrolabe) from orbit. It's the only way to be sure.

    ReplyDelete
  44. It was my understanding that Facts were not patentable or copyrightable.

    ReplyDelete
  45. @Waldo: intellectual properties(IP) are also facts, facts(discovery) or fiction, it's the source(idea/information) which is patentable or copyrightable.

    ReplyDelete
  46. "that while some data may have been checked against the ACS data, what has been 'copied' has also been corrected? Perhaps a 'bugs list' with the tz corrected data would be a useful document for the court case? The more evidence that the tz database IS the more accurate work would mitigate against the problem of a straight copy? "

    Copyright protects the expression of an idea, not the idea itself. ---That is the only way to express that idea-- is a defense, as well as -- I derived that expression of the idea on my own, independently of the other person's expression thereof-- (See the case of Phoenix Bios and the IBM PC where Phoenix created a chinese wall with one group of people reading the IBM bios, abstracting the ideas and required actions, and another group who had never read the IBM bios coding those requirements into a bios.) --I read and changed his expression of the idea, eg, correcting mistakes-- is not a defense, it is a derived work which still falls under the original copyright (and your copyright for the changes).

    ReplyDelete
  47. I'm actually shocked that revenue making companies actually use such an unreliable resource for something so important.

    ReplyDelete
  48. The point is: it is reliable! Olson and Eggert really put their energy into making sure there was no mistake.

    ReplyDelete
  49. You really should get Astrolabe's software and have a look at it. Probably the most beauty has is included in the Imperial Astrologer 1.0.6 (I guess about 25 MByte of brilliance). I got my copy! Get yours.

    Oops. I lost my copy.


    No panic! Got another one ...

    ReplyDelete
  50. Next thing it'll turn out time zones are patented Oo.

    ReplyDelete
  51. They deleted all the protestation posts from their Facebook fanpage and blocked people from writing new posts.

    Time to use this new Facebook feature : links can be posted into comments too and made quite visible :-)

    ReplyDelete
  52. From the Slashdot URL below:

    Shanks did not create the information in his published books, he collected it from other published sources, though the books fail to name the source of the information in nearly all case.

    Relevant sources of Shanks, especially for International atlas, were
    - Gabriel, Traite de l'heure dans le Monde, Paris
    - Henri Le Corre, Regimes Horaires pour l'Eruope et l'Afrique
    - Doris Chase Doane, Time changes in the world
    - Doris Chase Doane, Time changes in the USA
    - Curran and Taylor, World daylight saving time, Chicago 1935
    many other publications appearing in the astrology domain.

    http://slashdot.org/story/11/10/06/1743226/Civil-Suit-Filed-Involving-the-Time-Zone-Database

    ReplyDelete
  53. Ahhhhg 2012, the end of te world is here. Just the beginings...

    ReplyDelete
  54. Astrologers should be banned for anti-scientific heresy which poisons people's minds.

    ReplyDelete
  55. What they deserve:

    while [ 1 ]; do wget -m http://alabe.com/; rm -rf alabe.com; done

    ReplyDelete
  56. Well, I think we have to fight this absolute nonsense. If Astrolabe prevailed in this case, which they will not, because their claim is ridiculous, douchebag companies all around the world would try to do the same thing with other opensource projects.

    If you want to set up a mirror yourself, to help spread around the data all over the world, have a look here:

    http://github.com/canbuffi/tzmirror

    ReplyDelete
  57. Earler versions of the ACS Atlas did cite their sources, but later copies apparently expunged this data. The 1978 edition shows the following acknowledgements:

    Major contributions have been made by the following published materials and journals: Doris Chase Doane, "Time Changes in the USA"; Curran and Taylor, "World Daylight Savings Time"; documents published by the Interstate Commerce Commission; National Railway Publication Co.; "The Official Guide of the Railways ... of the United States..."; "The Mercury Hour"; "AFA Bulletin"; and "Astro Psychological Problems".

    It goes on to state... "Individuals making major contributions include: Mark Pottenger, Mary Brandes, Don Borkowski, Eugene Seger, Wayne Lee Holt, Mary Frances Wood and Arthur Blackwell. Significant material was also provided by Dorothy Pierce, John Adkins, Lillian Celic, Edith Custer, Gary Duncan, Lynne Burmyn, Liz Anderson, and Susan Aiu.

    I'd post the actual proof, but I might get sued.

    This is important though because if it turns out that ACS actually copied other peoples work from other compiled sources...

    Thanks to Dr. Alois Treindl for letting me know about this.

    ReplyDelete
  58. If you can't leave a message to these nice Astrolabe people on facebook then perhaps the good old fashioned email way will suffice. Let's send them some emails of appreciation to the following addresses - like one every minute :)

    astrolabe@alabe.com,
    support@alabe.com
    orders@alabe.com
    webmaster@alabe.com

    ReplyDelete
  59. To be clear, I and this blog do not support or condone any form of illegal direct action against Astrolabe.

    ReplyDelete
  60. Curtis Manwaring11 October 2011 at 00:23

    @Stephen Colebourne above - Exactly. The legal avenues will be enough and in the end Astrolabe will completely own this shame.

    ReplyDelete
  61. I'm not sure why people who make a living from lying to other people about "the stars" should be allowed the protection of the courts, or indeed food and water.

    Leaving aside that the threat has come from professional fraudsters, why is it that an accusation like this is so damaging? Why is it so easy to take down someone with a baseless accusation and why are there not more punishments for frivolous or malicious prosecution?

    ReplyDelete
  62. Hi Stephen,

    I found this page, and just checked in a copy of windows7.

    MS ship the entire TZ database in the registry of every copy of windows.......

    http://unattended.sourceforge.net/timezones.php

    Which points to a registry key.......

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time Zones

    And that has the TZ values in plain text..... Astrolabe should take on Microsoft.

    ReplyDelete
  63. What gets me is not the stupidity of these astrologists - but the stupidity of the US Judge who actually ordered the TZ database shut down simply on the basis of the dubious complaint. Couldn't he have at least waited until the matter was finally decided?

    Makes me wonder if the guy is competent to be a judge.

    ReplyDelete
  64. all russian "links to this post" from Roem.ru are actually spam.

    ReplyDelete
  65. Took me a while to google back the tz database from IANA. Hopefully there are alternatives such as timezonedb.com which provides CSV and SQL format.

    ReplyDelete
  66. "I'm not sure why people who make a living from lying to other people about "the stars" should be allowed the protection of the courts, or indeed food and water."

    OK - so people you don't agree with should not only be denied legal representation, but the right to live.

    I'm sure that works just fine in North Korea, but in most civilised countries, legal protection and the right to life are extended to everyone, even people you don't agree with.

    ReplyDelete