The Map Is Not The Territory

A blog by Christian Willmes.

New SemanticMediawiki based OSGeo Member Map

| categories: webdev, semantic web, geospatial, osgeo, semantic mediawiki | View Comments

In this post, I give some background on the new Semantic Mediawik based OSGeo Members map, that replaced the userMap. Starting with the Mediawiki update and introducing Semantic Mediawiki, some words about the history of the userMap and most important an overview of the new implementation and possible additional applications of Semantic Mediawiki in the OSGeo Wiki are given.

The introduction of Semantic Mediawiki into the OSGeo Wiki

Recently, thanks to an effort by OSGeo SAC (namely by Martin Spott), the OSGeo Wiki underlying Mediawiki software was upgraded from an ancient version (I think it was 1.12) to the current 1.25.3. Additionally the Semantic Mediawiki (SMW) extension, including Semantic Maps was installed, to enhance the OSGeo Wiki with its features.

SMW is a Mediawiki extension, that allows to structure wiki content (as data) and provides tools for queriying, export and visualization of this structured data. The Semantic Maps extension adds the capabilitiy to visualize SMW content, containing data of the special type "Geographic Coordinate" on maps. SMW even offers an API that allows to query the structured data stored in the wiki from external applications and export data based on queries. SMW is a mature project running on many large Mediawiki implementations, by well known organizations like NASA, OLPC, The Free Software Directory, semanticweb.org, to name just a few.

The OSGeo Wiki userMap

The original OSGeo Wiki userMap, implemented by me in 2008 during an internship at WhereGroup, is now broken because of dependencies of the not anymore supported Mediawiki extension called Simple Forms. The extension implemented a parser hook, that allowed to store the spatial locations of users in a PostGIS database. And parser hooks for including OpenLayers based map into wiki pages, displaying a users Location as well as a map of all were implemented in this first version of the userMap. The now deprecated documentaion is for now still available in the wiki, to get an overview.

SMW based OSGeo Members map

The SMW data model was developed using a tool called mobo. Due to using mobo, it is possible to develop and maintain an SMW data model from a central point in a consistent manner, enhancing maintainability, coordinating possible collaboration and also allowing to grow the Schema to additional applications and scopes over time. Mobo is a command line toolset that helps building Semantic MediaWiki structure in an agile, model driven engineering (MDE) way. The Schema is formulated applying the JSON-Schema specification, in JSON or YAML notation, in a defined folder structure considering file naming conventions. A bit similar to some MVC frameworks for building a web applications domain. The documaentation including a tutorial and examples of the mobo toolkit, can be found here.

The development code files of the mobo model are stored and published in a GitHub repository, for community review and allowing anyone to send pull requests for helping to improve the SMW based capabilities of the OSGeo Wiki.

It was even possible, to save the locations entered through the previous userMap implementation into the mentioned PostGIS table. This was possible by exporting the data from the PosGIS table as CSV, applying some Python foo on the CSV (especially on the geometry wkb notation using Shapely) and importing the data into the wiki as CSV, using the Mediawiki DataTransfer Extension.

Conclusion and Outlook

The application of SMW technology in the OSGeo wiki has, with the introduction of the OSGeo Members model, created a valuable directory that gives a nice overview of the OSGeo community. It is possible to extend the model in the future, to a directory of Charter Members, or OSGeo Advocates. This would yield sortable tables and of course maps of these contacts.

It is even possible to develop models for the Service Providers, to replace the sometimes hard to maintain current Service Provider directory, or for example a model of the Geo4All laboratories to generate directory and an according map. But one of my favorite possible models would be a model for an Open Geo Data directory in the OSGeo wiki.

All these models and the emerging directories would be collaboratively created and maintained by the OSGeo community by just editing the wiki. And not yet to speak of what is possible with the Mediawiki API for querying the structured data and getting the results nicely in JSON format, and by far not yet to speak of enabling the SPARQL-Endpoint which comes with Semantic Mediawiki.

So, the OSGeo Wiki has a bright future If we want. I will do my best for this goal.

Have fun!


 

comments powered by Disqus

Read and Post Comments

Misleading Chromium and Google Chrome warning message for self signed SSL certificates

| categories: webdev, ubuntu, open source, server | View Comments

Since some time, I am aware of a new Chromium or Google Chrome web browser warnig message concerning not "trusted" SSL certificates for SSL secured websites.

I guess that this new warnig message is part of Google's (basicaly good) campaign to support and promote the use of SSL. Google announced this campaign under the title HTTPS Everywhere at Google I/O 2014. They are talking about things like good citizenship of the web in the context of SSL.


Screenshot of the Chromium "Privacy error" warning, shown on accessing my own server via HTTPS.

The problem with that message is, that colleagues and freinds with whom I want to share data through my server, get scared by this misleading message from Chromium and Chrome, they get back to me saying that there might be somethig not working or wrong with my server. Then I have to try to explain to people, mostly barely knowing the difference between a website and a server, about SSL certificates and HTTPS, and convince them to trust me and not that serious appearing message... This does basically only work for people I know a fair bit. Some people with whom I need to work, but not know, will most probably not trust me and are scared away by this message, if they don't know enough about the matter of SSL encrypted HTTPS. And sorry Google, this is not good.


Even more misleading message chown by Chrome/Chromium if the user proceeds through the "Advanced" option.

In my view this warning message is not just very suggestive, in a way that it compromises the trust in accessing data and web applications on my server through HTTPS, it is also wrong in the content it claims. It says that accessing my server is unsafe. Which is not the case! And anybody who thinks that is the case when using a self signed certificate, please comment to this post and educate me.

I have now issued a free SSL certificate from StartSSL for the HTTPS configuration of my webserver, to get rid of this wrong and annoying warning by Chrome/Chromium. Which I am very uncomfortable with, because I do not trust this company in any way. And why the heck should I or anyone? I do not know anything about the people behind this company. And why the heck should I care? I just want to have a minumum protection for entering passwords and data into my webapplictation by providing HTTPS connection to my server. Since the Snowden revelations it is clear that SSL can be decrypted by knowledgeable enough "agency" anyway.... None the less, I am forced to trust in some company, which sells trust (which is plain wrong on so many different levels of implementation and from so many different angles of view on that matter). And I also need to force my colleagues and friends to trust in this company, from which I got some trust... This trust I gained throgh receiving and confirming an email send to an address on my domain name. That I host my Email not on the server, the domain is registered for, and where I use that certificate, does not matter for that company to trust me... :P.

On a side note, the warnings issued by FireFox or IE are way more polite, and do not scare away people from accessing my server (using a self signed certificate), they just accept the "asumed" and way less severe risk warnigs of those browsers notifications.

Finally, I have a question to you all. Please tell me, how a Self Signed Certificate is in any way less secure, than a "certified" and "trusted" one? The connection itself is not more or less secure, its just the trust. And as said, I am not comfortable with trusting some companies who can grant (sell) trust... This trust must come from the provider of the application and maintainer of the server that is to be accessed, I think.

Have fun and a good start into 2015!


 

comments powered by Disqus

Read and Post Comments

GeoNode installation on two hard disks

| categories: webdev, ubuntu, open source, geospatial, server | View Comments

I had to install GeoNode on a server with a (small) hard disk for the OS (Ubuntu Server) + software, and a second larger hard disk volume for the data. If you install geonode from the package source via apt-get, like me, you need to adapt the data locations to use the large hard disk volume. Otherwise, the data will end up on the small hard disk, where the GeoNode application is installed by default.

Because I had some work to find the best way on configuring the system in such an environment, I thought it would be good to write it into the internet, so that other people searching for solutions can find some.

System setup

As already mentioned the server runs a Ubuntu Server 14.04 LTS OS. GeoNode is installed via package install:

$ sudo add-apt-repository ppa:geonode/testing
$ sudo apt-get update
$ sudo apt-get install geonode

The second larger hard disk is mounted into '/media/data'. The goal is to have at least the most locations, where geonode stores its data on this volume.

Solution

According to an equiry on the geonode-users email list (thanks for helpfull answers to Ariel and Matthew), the following locations store the data, and will grow big over time.

/var/www/geonode/uploaded/
/var/lib/postgresql
/usr/share/geoserver/data/
/tmp/tmocat7

In the following, a solution for each of these locations is given.

GeoNode/GeoServer data directory

$ sudo mkdir /media/data/geoserver
$ cd /media/data/geoserver
$ mkdir data
$ cd /usr/share/geoserver
$ sudo ln -s /media/data/geoserver/data/ data
$ sudo chown tomcat7:tomcat7 data -R

Upload directory

$ cd /var/www/geonode
$ sudo mv uploaded /media/data/geonode/
$ sudo ln -s /media/data/geonode/uploaded/ uploaded
$ sudo chown www-data uploaded -R

Tomcat temp (cache) dir

Tomcat will write the cached tiles of GeoNode's GeoWebCache instance, which can get very big, into the tomcat temporary folder. The path of the temporary directory is defined in an environment variable, which is configured in the tomcat init/startup script.

$ cd /etc/init.d
$ sudo nano tomcat7

Find the line

JVM_TMP=/tmp/tomcat7-$NAME-tmp

...change it to

JVM_TMP=/media/data/tmp/tomcat/$NAME-tmp

Postgresql tablespace

The hardest part of the configuration is to change the file system locations of the postgresql database and its tables. At first we create a directory for the postgresql storage.

$ vmadmin@geonode:/media/data$ mkdir postgresql
$ sudo chown postgres postgresql/ -R
$ cd postgresql/
$ mkdir data
$ sudo chown postgres data/ -R

Next we set the table spaces:

$ sudo su postgres
$ psql
CREATE TABLESPACE exthd LOCATION '/media/data/postgresql/data';

ALTER DATABASE geonode SET default_tablespace = exthd;

\connect geonode

Alter the tables that grow big to new tablespace:

ALTER TABLE documents_document SET TABLESPACE exthd;
ALTER TABLE layers_attribute SET TABLESPACE exthd;
ALTER TABLE layers_layer SET TABLESPACE exthd;
ALTER TABLE layers_layer_styles SET TABLESPACE exthd;
ALTER TABLE layers_layerfile SET TABLESPACE exthd;
ALTER TABLE layers_style SET TABLESPACE exthd;
ALTER TABLE layers_uploadsession SET TABLESPACE exthd;
ALTER TABLE maps_map SET TABLESPACE exthd;
ALTER TABLE maps_maplayer SET TABLESPACE exthd;
ALTER TABLE maps_mapsnapshot SET TABLESPACE exthd;
ALTER TABLE services_service SET TABLESPACE exthd;
ALTER TABLE services_servicelayer SET TABLESPACE exthd;
ALTER TABLE services_serviceprofilerole SET TABLESPACE exthd;
ALTER TABLE services_webserviceharvestlayersjob SET TABLESPACE exthd;
ALTER TABLE services_webserviceregistrationjob SET TABLESPACE exthd;
ALTER TABLE upload_upload SET TABLESPACE exthd;
ALTER TABLE upload_uploadfile SET TABLESPACE exthd;

The tables are pure assumption, its possible to alter the tablespace of more tables later on, if further tables proof to store much data. Thats it, so far... until now everything runs as expected on the system, and the right locations store the data. I'll keep you posted if I need to adapt anything on the system.

Have fun!


 

comments powered by Disqus

Read and Post Comments

SLD production for use with GeoNode/GeoServer

| categories: webdev, geospatial, open source | View Comments

The Problem

Normally I use QGIS as a Desktop GIS for producing geodata and visualization of it for the deployment of web services. For MapServer WMS there is a great toolchain by using the MapServer export plugin. But since a while I am working with the even greater GeoNode application for publishing geodata, which uses GeoServer for the deployment of OGC Services (including WMS).

From a first look, there is no problem, because QGIS supports SLD export out of the box, and GeoNode / GeoServer accepts SLDs for the styling of WMS services. But sadly the SLD produced by QGIS is version 1.1.0 and the GeoNode / GeoServer only accepts SLD 1.0.0 at the moment.

<StyledLayerDescriptor xmlns="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/sld http://schemas.opengis.net/sld/1.1.0/StyledLayerDescriptor.xsd" xmlns:se="http://www.opengis.net/se">
QGIS generated SLD 1.1.0 header.

A solution

So I was looking around for a tool which can generate and edit SLD 1.0.0 conformant styles. I looked into other GIS desktop applications I have at hand. I did not even dare to think that Arc* does support interoperable styles... indeed it does not. ;-)

But AtlasStyler to the rescue! AtlasStyler is a nice small Java application which offers an intuitive GUI for editing SLD Styles, and best is, it produces SLD 1.0.0.

Screenshot of AtlasStyler GUI.

Additionaly, If you have some basic CSS knowledge, the created SLD file can be easily adjusted in a text editor of your choice. This is maybe needed for some more complex SLDs. I had to adjust for example the stroke width of the geometry outlines, because the AtlasStyler GUI only allows natural number (integer) values for the stroke-width parameter.

<sld:StyledLayerDescriptor xmlns="http://www.opengis.net/sld" xmlns:sld="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml" version="1.0.0">
AtlasStyler generated SLD 1.0.0 header.

Hope this little work around for QGIS -> GeoNode data publishing toolchain is of use for the one or the other around...

Have fun!



 

comments powered by Disqus

Read and Post Comments

Modelling bibliographic records in Semantic MediaWiki using BibTeX schema and result format

| categories: webdev, semantic web, semantic mediawiki | View Comments

Disclaimer: This is a bit longish post about modelling Bibliographic Information in Semantic Mediawiki.

Semantic MediaWiki (SMW) supports to manage bibliographic records and deliver them in BibTeX format by using the Semantic Result Format BibTeX. This is a useful feature, if you understand how to implement it in your SMW instance, which is not trivial if you are not already an SMW expert. In this post I try to describe this modelling and implementation process.

Ok this last paragraph and the headline of this post contain a lot of maybe new information (for non SMW experts), which needs to be clarified first.

  • Bibliographic Record
  • BibTeX Format
  • SMW Semantic Result Formats
  • BibTeX Semantic Result Format

Bibliographic Record

A bibliographic record is an entity to reference a specific content item, which is in most cases an academic publication, for example a journal paper. Those bibliographic records mostly underlie a schema or formalism which is applied in a given context, for example references in an academic publication mostly follow a citation formalism defined by the publisher.

BibTeX Format

The BibTeX Format is a tool to model such citation formalisms, originating from the LaTeX community, to handle bibliographic records in LaTeX. Though, there does not exist any official specification of the BibTeX schema (aside from the BibTeX implementation in the LaTeX code base), but in the following we refer to the Wikipedia entry, which defines the schema in a sufficient way.

Semantic Result Format

Semantic Result Formats (SRF) is a SMW extension which allows to render the results of an SMW #ask query or inline query in a defined format.

BibTeX SRF

The BibTeX SRF allows to render bibliographic information, stored in an SMW instance, in BibTeX Format. Here are some demos of the BibTeX SRF.

Modelling BibTeX schema in SMW

In the mentioned Wikipedia article, the BibTeX schema is defined in bibliographic items, which are the basic attributes or properties of bibliographic entry types or classes.

Bibliographic Items

The bibliographic items are modelled as SMW properties. The BibTeX Wikipedia site defines 26 items, to which we add three more items. (1) keyword, to handle the keywords defined for the content of the publication as semantic properties. This has the advantage, that you can browse and filter for keywords in the constructed bibliographic database. And we define a property for (2) DOI and (3) ISBN, which are two well accepted unique identifier schemes for publications. This gives us the following list of bibliographic items:

  • address: Publisher's address (usually just the city, but can be the full address for lesser-known publishers)
  • annote: An annotation for annotated bibliography styles (not typical)
  • author: The name(s) of the author(s) (in the case of more than one author, separated by and)
  • booktitle: The title of the book, if only part of it is being cited
  • chapter: The chapter number
  • crossref: The key of the cross-referenced entry
  • DOI: Digital Object Identifier (www.doi.org)
  • edition: The edition of a book, long form (such as "First" or "Second")
  • editor: The name(s) of the editor(s)
  • eprint: A specification of an electronic publication, often a preprint or a technical report
  • howpublished: How it was published, if the publishing method is nonstandard
  • institution: The institution that was involved in the publishing, but not necessarily the publisher
  • ISBN: International Standard Book Number
  • journal: The journal or magazine the work was published in
  • key: A hidden field used for specifying or overriding the alphabetical order of entries (when the "author" and "editor" fields are missing). Note that this is very different from the key (mentioned just after this list) that is used to cite or cross-reference the entry.
  • keyword: Keyword(s) to tag/categorize the content of the publication
  • month: The month of publication (or, if unpublished, the month of creation)
  • note: Miscellaneous extra information
  • number: The "(issue) number" of a journal, magazine, or tech-report, if applicable. (Most publications have a "volume", but no "number" field.)
  • organization: The conference sponsor/host
  • pages: Page numbers, separated either by commas or double-hyphens.
  • publisher: The publisher's name
  • school: The school where the thesis was written
  • series: The series of books the book was published in (e.g. "The Hardy Boys" or "Lecture Notes in Computer Science")
  • title: The title of the work
  • type: The field overriding the default type of publication (e.g. "Research Note" for techreport, "{PhD} dissertation" for phdthesis, "Section" for inbook/incollection)
  • url: The WWW address
  • volume: The volume of a journal or multi-volume book
  • year: The year of publication (or, if unpublished, the year of creation)

You are free to extend this list with any item you want or which you think would be useful. For example an citation item, in which you store the complete Citation, as you would add it in a Bibliographic reference list at the end of an publication. I use the note item for this purpose, but...

Entry Types

The entry types are modelled as SRF classes holding the according properties (bibliographic items) in SMW. According to the Wikipedia BibTeX scheme we have 14 entry types, of which I show here the five most used:

Entry Type Description Required Items Optional Items
article An article from a journal or magazine. author, title, journal, year keywords, volume, number, pages, month, DOI, URL, note, key
book A book with an explicit publisher. author/editor, title, publisher, year keywords, volume, number, series, address, edition, month, ISBN, URL, note, key
inbook A part of a book, usually untitled. May be a chapter (or section or whatever) and/or a range of pages. author/editor, title, chapter/pages, publisher, year keywords, volume/number, series, type, address, edition, month, ISBN, URL, DOI, note, key
inproceedings An article in a conference proceedings. author, title, booktitle, year keywords, editor, volume/number, series, pages, address, month, organization, publisher, DOI, URL, ISBN, note, key
techreport A report published by a school or other institution, usually numbered within a series. author, title, institution, year keywords, type, number, address, month, DOI, URL, note, key

Implementation

For implementing the data structure in SMW, we use the Semantic Forms extension. Semantic Forms facilitates GUI's to create and edit structured data in SMW. Basically it allows users to add, edit and query data in SMW using forms.

The easiest way to implement the bibliographic data model is to use the Semantic Form "Create a Class". This creates all properties, forms, and templates automatically by filling out a form.

Screenshot of the "Create a Class" form, defining the BibBook class.

After filling out the Form and clicking on "create", you need to go to Special:SMWAdmin and run the "Start updating data", this triggers SMW to create all needed links, so you can find and work with the Forms and Templates in your wiki.

You can repeat this create class process for each Entry Type you want to implement. You need to enter all the bibliographic item properties again, so that the Forms and templates will contain them. The bibliographic item properties will not be duplicated if they already exist in SMW though.

Semantic Forms

Using the create class form SMW automatically created templates and forms to display and edit the data of the according class. The automatically created forms are fine, but with two minor edits you don't have to specify the Entry Types for each new item, which would be redundant, because we already defined the entry type through the class definition. In example we edit now the template Template:BibArticle and the form Form:BibArticle of the BibArticle class, to set the Entry Type automatically.

From the Form, we remove the

! BibType:
| {{{field|BibType}}}
|-
part, which would let the user enter a value for the BibType property, which we do not want in our model. The resulting form definition looks as follows:

Form:BibArticle

<noinclude>
This is the "BibArticle" form.
To create a page with this form, enter the page name below;
if a page with that name already exists, you will be sent to a form to edit that page.

{{#forminput:form=BibArticle}}
</noinclude><includeonly>
<div id="wikiPreview" style="display: none; padding-bottom: 25px; margin-bottom: 25px; border-bottom: 1px solid #AAAAAA;"></div>
{{{for template|BibArticle}}}
{| class="formtable"
! Author(s):
| {{{field|Author(s)}}}
|-
! Title:
| {{{field|Title}}}
|-
! Journal:
| {{{field|Journal}}}
|-
! Year:
| {{{field|Year}}}
|-
! Volume:
| {{{field|Volume}}}
|-
! Number:
| {{{field|Number}}}
|-
! Pages:
| {{{field|Pages}}}
|-
! Date:
| {{{field|Date}}}
|-
! DOI:
| {{{field|DOI}}}
|-
! URL:
| {{{field|URL}}}
|-
! Keyword(s):
| {{{field|Keyword(s)}}}
|-
! Key:
| {{{field|Key}}}
|-
! Note:
| {{{field|Note}}}
|}
{{{end template}}}

'''Free text:'''

{{{standard input|free text|rows=10}}}

{{{standard input|summary}}}

{{{standard input|minor edit}}} {{{standard input|watch}}}

{{{standard input|save}}} {{{standard input|preview}}} {{{standard input|changes}}} {{{standard input|cancel}}}
</includeonly>

In the template we set the BibType property statically, so that every BibArticle is of BibType::Article, we set [[BibType::Article]] as first entry. Additionally we set the category "Bibliographic Record" for the entry (last line), because every BibArticle is a Bibliographic Record. So you can later query for example for all Bibliographic Record's, yielding different entry types. See the following Template definition code:

Template:BibArticle

<noinclude>
This is the "BibArticle" template.
It should be called in the following format:
<pre>
{{BibArticle
|BibType=
|Author(s)=
|Title=
|Journal=
|Year=
|Volume=
|Number=
|Pages=
|Date=
|DOI=
|URL=
|Keyword(s)=
|Note=
|Key=
}}
</pre>
Edit the page to see the template text.
</noinclude><includeonly>{| class="wikitable"
! BibType
| [[BibType::Article]]
|-
! Author(s)
| {{#arraymap:{{{Author(s)|}}}|,|x|[[BibAuthor::x]]}}
|-
! Title
| [[BibTitle::{{{Title|}}}]]
|-
! Journal
| [[BibJournal::{{{Journal|}}}]]
|-
! Year
| [[BibYear::{{{Year|}}}]]
|-
! Volume
| [[BibVolume::{{{Volume|}}}]]
|-
! Number
| [[BibNumber::{{{Number|}}}]]
|-
! Pages
| [[BibPages::{{{Pages|}}}]]
|-
! Date
| [[BibDate::{{{Date|}}}]]
|-
! DOI
| [[BibDOI::{{{DOI|}}}]]
|-
! URL
| [[BibURL::{{{URL|}}}]]
|-
! Keyword(s)
| {{#arraymap:{{{Keyword(s)|}}}|,|x|[[BibKeyword::x]]}}
|-
! Note
| [[BibNote::{{{Note|}}}]]
|-
! Key
| [[BibKey::{{{Key|}}}]]
|}

[[Category:BibArticle]]
[[Category:Bibliographic Record]]
</includeonly>

Here you can find further examples and the sources of more Entry Type definition.

Authoring and editing bibliographic data

All authoring and editing is facilitated by the Forms we have created for the entry type classes. You can create new entries as well as editing existing entries using those forms.

Screenshot of the form for editing BibArticle entries.

Conclusion

In this post, the implementation of a bibliographic model in SMW was described in detail. You can find this implementation in my SMW instance, where you can look at the details I may forgot to mention here.

The actual use of the SMW based bibliographic data base will be described in an upcoming blog post soon. There I will dig into the powerful browsing, filtering and data rendering capabilities of SMW.

I hope this post helps some people getting their heads around the SMW concept, which can be kind of complex... As I heard of SMW first, it was immediately clear to me that this is a very powerful technology, which makes much sense. But I had to chew a bit on all of the concepts before it worked for me (after much of trial and error)...

Have fun!


 

comments powered by Disqus

Read and Post Comments

Next Page ยป