Samuel's Project Blog

With the release of RDFIO 0.4.0, my GSoC 2010 project is now over!

I want to thank especially my mentor Denny Vrandečić, and also the SMW community at large for a great time! I also want to sincerely thank my masters project mentor Egon Willighagen who mentioned about, and encouraged me to apply to the program. Without this encouragement, I'd never taken the step. It has been a good time and rewarding, and I've much enjoyed to have time to get a bit into MediaWiki/SMW extensions development, as well as to provide some new functionality to these great bits of software.

The main GSoC coding is now over, and I will need to take a little break for an exam this friday, but surely I'll continue to refine the RDFIO extension later, especially as Egon and me are looking into using it to integrate Bioclipse with SMW in order to make RDF data in Bioclipse "Community editable" (could turn out to be some real useful stuff!).

The 0.4.0 release

This new release brings a lot of refactoring and reworkings under the hood, as well as quite a few minor bugfixes and improvements here and there, so upgrading is recommended. We also got the issues in SMWWriter fixed now, so patching it is no longer needed, which hopefully will make installing easier!

• Se the extension page for more info about download and install

Of the more notable changes are the improved selection of wiki titles on import, as described in this blog post. Another important fix is that the default output format (if not specifying any) is now "SPARQL Resultset XML" which now makes the SPARQL endpoint fully "SPARQL compliant" and queryable from typical SPARQL tools like Jena. It is a remaining topic though how to allow update operations without leaving the endpoint wide open ... i.e. how to implement some form of user rights checking, when used as a webservice.

A little technical note also is that RDFIO now takes the $wgDBprefix parameter into account, so if you are using RDFIO with table prefixes in the database, you will need to regenerate the tables and the triples in the store (can be done at the Special:ARC2Admin and Special:SMWAdmin pages respectively).

I should not end without a note about some great bits of existing code that I've had the pleasure to make use of:

• ARC, the PHP RDF library I've been using, and on which RDFIO is very heavily dependent
  I found it very powerful and super conveniant to work with!
• I enjoyed making use of the SMWWriter and PageObjectModel extensions, which also definitely made life easier for me and saved me tons of work.

This week I just finished the last remaining items on my todo list for my Google Summer of Code project, (which is available in the form of the RDFIO MediaWiki extension). Those things, which I also mentioned in my last blog post were to:

• Add ability to use ("pseudo") namespaces for general RDF entities (non-properties) in order to choose wiki titles for them on RDF import.
• Add a screen that shows URIs lacking a namespace prefix to abbreviate it with.

Regarding the first point, it might not be overly easy to see the usefulness of it at once, so I just created a screencast to show the difference between using it and not:

• Link to youtube video page

It demonstrates the problem of choosing sensible wiki titles for general RDF entities in case no good property for naming is available, (such as rdfs:label etc) ... since "entity" URIs often just consist of nonsensible id:s and often no namespace prefixes are defined for them. RDFIO lets you add "pseudo" namespaces (using a simplified splitting pattern, not necessarily consistent with XMLns specs), in order to come around this problem.

• The new functionality is so far only available in the svn trunk
• More info, install instructions etc on the extension page

Hopefully I'll find time to also demonstrate the second point above, as well as the "filter by ontology" feature for the SPARQL endpoint, with screencasts early next week.

Otherwise, the coming week I'll use for doing some refactoring of the currently quite unmanageable code, as well as add commenting, and hopefully also add the feature to filter RDF export by a [[Export RDF::false]] SMW property (which was the "it time permits" item of my TODO list).

The filter by ontology / vocabulary feature

Just to inform that I created a new release of the RDFIO MediaWiki extension. It contains important security fixes, by adding at least some basic checking of user rights and CSRFs (Cross site request forgeries) to the SPARQL endpoint, RDF import form etc. Thus, it's highly recommended to upgrade if you are using the extension on a public wiki!

• Download tarball
• Svn repo url: http://smwrdfio.googlecode.com/svn/branches/0.2.x
• Install instructions

Also, you might already have seen:

• The demo screen cast
• The install screencast

Current GSoC status

Otherwise, me and Denny just confirmed the remaining TODO list for my GSoC project, which is what I start working on now:

1. In the SPARQL endpoint, enable querying by any URI specified as equivalent URI
2. For RDF export, implement an "export by ontology" option, that - when possible - restricts the URIs used for a wiki page to only those that appears in an ontology that the user points to
3. Create an HTML interface for interactively configureing how wiki titles should be chosen for RDF entities for which no preferred "wiki title property" (such as rdfs:label, dc:title etc.) was found.
4. Add "pseudo namespaces" as an option for choosing wiki titles from general RDF URIs (not only properties!). I.e, the possibility to abbreviate a part of an URI into a pseudo-namespace, making the URI more fit for use as wiki title. (For properties, there if often a well known abbreviation for the corresponding vocabulary/ontology's base URI, but this is often not the case for general RDF entities, which can often be from some user defined data etc).

There are also some extra additions that I'll look into if time permits, like adding support for filtering the output on export with a property such as "RDF export::False", as suggested by Daniel Herzig.

In a previous blog post I demonstrated with a screen cast the RDFIO extension for Semantic MediaWiki but nothing on installation.

By testing I realized that the install procudure was VERY painful. I have now (with much valuable help from Oleg Simakoff) corrected a number of errors in the instructions and the code, and added to the install instructions commandline snippets for linux/ubuntu. I also created a screencast which goes through the steps from scratch (except Apache/MySQL/PHP setup), in a little more than 5 minutes. Hope this makes things easier for you testers! (And as you might try it out, please report any bugs or issues in the issue tracker!)

Sorry for the low volume level! Didn't realize that while recording ... :/

So, for those of you who might think the Install instructions for the RDFIO Semantic MediaWiki extension I'm working on are a bit daunting but would like a glimpse of what my GSoC project is up to anyway, I created a short (3:20) screencast demonstrating (ARC2 based) RDF Import and SPARQL "Update" functionality for some example data. (Sorry for the lame speaking ... :P ... didn't sleep for a looong time )

The screencast shows how you can import RDF/XML into Semantic MediaWiki and then use the SPARQL endpoint to insert or remove data to/from articles, even using the original format of the RDF that you imported earlier.

(For you who decide to try to install, please have a look at the error fixing happening in this thread.)

I just moved to a new Google code repository, reflecting the name change of the MediaWiki extension from "SMW RDF Connector" (it's awfully long, isn't it) to "SMW RDFIO", or just "RDFIO", so please update your links!

• Main repo url: http://code.google.com/p/smwrdfio
• Keep track of my check-ins: http://code.google.com/p/smwrdfio/source/list

See also the newly created extension page, which will be the hub for information about the extension in the future.

(Figured I better do regular status updates, as a lot of small things tend to get missed if blogging only when there is something to show off)

As you might have seen among the GSoC2010 tagged posts I've had a rudimental RDF/XML import, and a SPARQL endpoint (only for querying so far!) up running for a while. You should be able to set up these yourself by following one or more of the instructions in the Google code repo:

• ARC2 Store Install (required for SPARQL endpoint)
• ARC2 SPARQL Endpoint Install
• RDF Import Install

I have since worked a bit on some use cases, which revealed a lot of intricacies to take into account on RDF import. One of them was a spinoff discussion, from a blog post by Egon Willighagen, which quite nicely outlines one of the motivations for having general RDF import in MediaWiki (read post, read discussion).

The last few days I've been working on heavly refactoring the import code, so that it is more general and easy to modify in new ways. There is still a lot to be improved in the code, like error handling, documentation, adding more options etc, so feel free to give feedback on the code! (Especially RDFImporter.php and EquivalendURIHandler.php, and preferrably use the mailing lists: semediawiki-devel, semediawiki-user or mediawiki-l)

The RDF import seems to be the most challenging part in my project (and on which the export feature heavily depends) - since it is the part where I'm breaking a bit of new ground, so here feedback is much welcome.

Choosing wiki titles for RDF entities on import - Feedback wanted

The one most challenging issue is about how to select reasonable wikititles to use for RDF entities on RDF import, based on the RDF data (one relevant blog post here). The question of being able to export the page with the original URI, should not limit the choice directly, since this is already solved by storing the original URI as a property on each page.

The thoughts we have had so far - in short - is:

• First look if the RDF entity in question has one of a list of properties, in prioritized order, that should be used as wiki title.
  • The first of these, could be a special property which can be used to manually specify this by including it in the imported RDF, like "hasWikiTitle".
  • The suggested list or properties so far can be seen in this blog post. This list should of course be configurable, and one question is also how to best implement this configuration? A setting in LocalSettings.php? A wiki page?
• If no matching property is in this list, then the label for the RDF entity should be used. For example, if the entity:s URI is http://bio2rdf.org/go:0032283, then 0032283 is used.

How to configure namespace prefixes / preudo-namespaces?

Using only the label of course has the risk that multiple RDF entities converts to the same wiki title, which is not acceptable for example if using the wiki as a "one time RDF editor", which is one of the motivations for this project.

To solve this, one alternative (as a configurable option) could be to use a pseudo-namespace in the wiki title (e.g. "go" in the above example, which would result in "go:0032283" as the wikititle). This could be configured by creating a mapping between base URI:s and pseudo-namespaces (.e. "http://bio2rdf.org/go:" and "go", in this case).

But then there is the question how to configure this mapping. We've been thinking of a few options:

• Let it be configured in the incoming RDF, by using a custom predicate "hasPseudoNameSpace"
• Store a config in a Wiki article
• Config in LocalSettings.php
• On submitting data for import, analyze it first and present a screen with all the base URI:s used, with fields to manually fill in the pseudo-prefixes to use.
• Any combination of the above

I will be working ahead, and try to figure out the most reasonable strategy together with Denny (who is my GSoC mentor), but feedback and comments are always welcome! (As said, preferrably send feedback on the mailing lists; semediawiki-devel, semediawiki-user or mediawiki-l!)

If you want to follow the project progress, see the status page for options

 I ran into some troubles with the debugging with XDebug in Eclipse for PHP Developers / PDT (breakpoints stopped to take / catch, after I changed location of my www folder - which I figured out later), so I wanted to document the full setup procedure it here. I mainly followed this blog post. (Assuming you have apache and php set up!).

• Download Eclipse for PHP developers
  • I'm using the Linux 32 bit version
  • Because of a bug with PDT for Eclipse Helios, I chose to go with Eclipse PDT SR2 for now.
• Install XDebug by typing:

 apt-get install php5-xdebug

• Edit /etc/php5/conf.d/xdebug.ini to look something like this:

 zend_extension=/usr/lib/php5/20090626+lfs/xdebug.so
 xdebug.remote_enable=On
 xdebug.remote_host="localhost"
 xdebug.remote_port=9000
 xdebug.remote_handler="dbgp"
 # xdebug.remote_log="/tmp/xdebug.log" 

• • The log line can be good to enable in case you run into troubles.
  • The first line can be different depending on PHP version. Use the line that you can find in the file (which is the default for your version of Ubuntu):

 /etc/php5/conf.d/xdebug.ini.ucf-dist

• Restart apache:

 sudo apache2ctl restart

Configure Eclipse

• Create a PHP Project. Mine is named "RDF4SMW".
• Choose Run > Debug Configurations in Eclipse
• Turn the configuration to your web server setup. For me, it looks like this ("smw" is the folder where MediaWiki is installed):

Custom location of www folder

Then an important thing If you have changed the location of your www folder from /var/www, then don't miss this! This was what caused the problem for me, with breakpoints not taking:

• Click the "Configure" button in the Debug configuration screen seen above
• Click the "Path mapping" tab
• Configure according to the location of your www folder. My current config looks like so:

Most probably, the extended RDF import/export in Semantic MediaWiki, which I'm working on, will be made an extension, to start with (at least that's my idea).

First I was calling it "RDFIO", but in fear of being a too general name, or too undescriptive, I switched to "SMW RDF Connector", and the project currently lives in a repo called SMW RDF Connector.

Now, realize this is awfully long, I'm thinking to go back to "RDFIO", or rather "SMW RDF IO" again, before it's "too late", (haven't even created a MediaWiki extension page yet). Seems like it's the most clever, short choice after all.

Any comments?