BlogForever

Recently I have been putting together my report on the extent to which the BlogForever platform operates within the framework of the OAIS model. Inevitably, I have thought a bit about migration as one of the potential approaches we could use to preserve blog content.

Migration is the process whereby we preserve data by shifting it from one file format to another. We usually do this when the “old” format is in danger of obsolescence for a variety of reasons, while the “target” format is something we think we can depend on for a longer period of time. This strategy works well for relatively static document-like content, such as format-shifting a text file onto PDF.

The problem with blogs, and indeed all web content, is when we start thinking of the content exclusively in terms of file formats. The content of a blog could be said to reside in multiple formats, not just one; and even if we format-shift all the files we gather, does that really constitute preservation?

With BlogForever, we’re going for an approach to capture and ingest which seems to have two discrete strands to it.

(1) We will be gathering and keeping the content in its “original” native formats, such as HTML, images files, CSS etc. At time of writing, the current plan is that we will have a repository record for each ingested blog post and all its associated files (original images, CSS, PDF, etc.) will be connected with this record. These separate files will be preserved and presumably migrated over time, if some of these native formats acquire “at risk” status.

(2) We are also going to create an XML file (complete with all detected Blog Data Model elements) from each blog post we are aggregating. What interests me here is that in this strand, an archived blog is being captured and submitted as a stream of data, rather than a file format. It so happens the format for storing that data-stream is going to be XML. The CyberWatcher spider is capable of harvesting blog content by harnessing the RSS feed from a blog, and by using blog-specific monitoring technologies like blog pings; and it also performs a complex parsing of the data it finds. The end result is a large chunk of “live” blog content, stored in an XML file.

Two things are of interest here. One is that the spider is already performing a form of migration, or transformation, simply by the action of harvesting the blog. Secondly, it’s migrating to XML, which is something we already know to be a very robust and versatile preservation format, more so even than a non-proprietary tabular format such as CSV. The added value of XML is the possibility of easily storing more complex data structures and multiple values.

If that assumption about the spider is correct, perhaps we need to start thinking of it as a transformation / validation tool. The more familiar digital preservation workflow assumes that migration will probably happen some time after the content has been ingested; what if migration is happening before ingest? We’re already actively considering the use of the preservation metadata standard PREMIS to document our preservation actions. Maybe the first place to use PREMIS is on the spider itself, picking up some technical metadata and logs on the way the spider is performing. Indeed, some of the D4.1 user requirements refer to this: DR6 ‘Metadata for captured Contents’ and DR17 ‘Metadata for Blogs’.

We anticipate the submitted XML is going to be further transformed in the Invenio repository via its databases, and various metadata additions and modifications will transform it from a Submission Information Package into an Archival Information Package and a Dissemination Information Package. As far as I can see though, the XML format remains in use throughout these processes. It feels as though the BlogForever workflow could have a credible preservation process hard-wired into it, and that (apart from making Archival Information Packages, backing-up and keeping the databases free from corruption) very little is needed from us in the way of migration interventions.

It also feels as though it would be much easier to test this methodology; the focus of the testing becomes the spider>XML>repository>database workflow, rather than a question of juggling multiple strategies and testing them against file formats and/or significant properties. Of course, migration would still need to apply to the original native file formats we have captured, and this would probably need to be part of our preservation strategy. But it’s the XML renditions which most users of BlogForever will be experiencing.

by Prof David Olson,

ERP has undergone a rapid and dramatic evolution.Since SAP started working on their accounting product in the early 1970s, five major and many smaller vendors dominated large organizational computing in the 1990s.Most large organizations took advantage of the integrative opportunities of this form of software to rely on COTS products to downsize their very large information technology staffs.Y2K created a boom period for both ERP vendors and for IT individuals, but after the world somehow survived COBOL’s minor limitations, the demand for BOPSE ERP systems dropped, and IT individuals found a very depressing job market.The first decade of the 21^st Century has seen highly cyclical IT employment, demonstrating the need to be mobile.The five BOPSE ERP vendors have collapsed to two (although both SAP and Oracle are very large and prosperous).There has been stronger emphasis on industry systems, support to small businesses, and country-specific software products.

Technologically, there have been many interesting developments in software-as-a-service, and open source software projects.SourceForge.net contains hundreds of thousands of open source software projects, including about 1,000 classified as ERP.These projects tend to be utilities in nature, similar to what software-as-a-service offers for a fee over the Internet.Countries such as Brazil are publicly supporting development of open source ideas.There are a growing number of ERP vendors offering their software for free (installation help and training available at a fee), such as Compiere and Nexedi.There even are a few efforts to develop a completely open source ERP product.

Software development is clearly evolving.My perception is that globally, there is strong preference for the open ideas of Richard Stallman in preference to the US-dominated proprietorial model of Bill Gates and Larry Ellison.The Internet enables enormous potential in linking active minds around the globe, enabling their collaboration in developing new and better things.

Looking closer to the Blogforever.eu project and ALTEC’s interest for commercial uptake and adoption, according to which the blogosphere could be a means to develop organizational computing, taking advantage of database technologies,  my first reaction is a disconnect in my own mind between ERP and database issues. However, I think that there is potential there.:  last fall I taught a database class, and had students report on databases of their selection.There were a number of very interesting reports of databases for various Web-businesses, such as Facebook, Craig’s List, Amazon, etc.The database systems to support things in the blogosphere certainly exist.Objects such as SaaS, open source software, and the apps that the latest generation love can be assembled to accomplish things that people and organizations need.You could conceivably assemble the software you need from a free and open Internet platform.

This is going to take a long time to evolve.But five years ago I thought an open source ERP software was impossible.It demonstrably is possible.There is a danger in forecasting, as anyone who reads the forecasts of the 19^th Century, or Aldous Huxley or George Orwell knows.It is interesting to review any forecast over 20 years old.They inevitably miss many important factors.But I do feel confident that the means for collaborative software product development can support such a free-form software environment.

A key issue is how the benefits will be shared, and how contributors will be rewarded.I think that recent times have been dominated by US ideas about such matters.I perceive that is changing radically, and for the better.After all, Bill Gates is looking for ways to redistribute all of his gains, isn’t he?

—

David L. Olson is the James & H.K. Stuart Professor in MIS and Chancellor’s Professor at the University of Nebraska.  He has published research in over 100 refereed journal articles, primarily on the topic of multiple objective decision-making and information technology.  He teaches in the management information systems, management science, and operations management areas.  He has authored 18 books, to include Decision Aids for Selection Problems, Introduction to Information Systems Project Management, Managerial Issues of Enterprise Resource Planning Systems, Supply Chain Risk Management, and Supply Chain Information Technology.  Additionally, he has co-authored the books Introduction to Business Data Mining, Enterprise Risk Management, Advanced Data Mining Techniques, New Frontiers in Enterprise Risk Management, Enterprise Information Systems, and Enterprise Risk Management Models. He is associate editor of Service Business and co-editor in chief of International Journal of Services Sciences. He has made over 100 presentations at international and national conferences on research topics.  He is a member of the Decision Sciences Institute, the Institute for Operations Research and Management Sciences, and the Multiple Criteria Decision Making Society.  He was a Lowry Mays endowed Professor at Texas A&M University from 1999 to 2001.  He was named the Raymond E. Miles Distinguished Scholar award for 2002, and was a James C. and Rhonda Seacrest Fellow from 2005 to 2006.  He was named Best Enterprise Information Systems Educator by IFIP in 2006.  He is a Fellow of the Decision Sciences Institute.

The third BlogForever Consortium Meeting will take place during 22-23 February 2012 in Berlin. All project partners will come together to discuss about current progress and plan for our next steps.

The main meeting topics will be:

• The recent outcomes of WP2 Weblog Structure and Semantics ( D2.4 Weblog Spider Prototype and Associated Methodology Report ) as well as next developments in WP2.
• Ongoing work on WP3 The BlogForever Policies
• The recent outcomes of WP4 BlogForever Software Infrastructure ( D4.1 User Requirements and Platform Specifications Report ) as well as next developments in WP4.
• Initial Planning for WP5, Case studies and validation
• Dissemination activities & market analysis (WP6).

Herbert van de Sompel will be our guest for this meeting. Herbert is the team leader of the Prototyping Team at the Research Library of the Los Alamos National Laboratory. Currently, he works with his team on the Open Annotation and Memento (time travel for the Web) projects.