I’ve been coming up to York Beach almost every summer for over 40 years.  This was the first time I had ever seen, or rather not seen, the light operating.  I did what anyone in 2010 does when something strange happens, I checked Google.  However after searching for news related to this “outage” I came up empty.  The Town of York website was silent on the issue as was a local news website.

The blackout continued on through Sunday and Monday.  On Tuesday morning we noticed that the light was back in operation.  Whether there was an electrical issue or a burned out light bulb, it was apparently resolved early Tuesday.  I sit here writing this particular paragraph on Tuesday evening with the light operating in its typical and (usually) reliable manner.  Maybe someday I’ll learn what happened.

Beyond the mystery of the lighthouse, it was an unusually warm week in York.  Upper 80s and lower 90s prevailed each day.  For several days there wasn’t much wind, at least not a cool wind, at the beach.  This is the first year in recent memory where Lisa hasn’t been on the beach wrapped in sweatshirts and towels to keep warm.  Complementing the hot weather, the water was about 63 degrees and provided a very refreshing respite from the heat!

The surf provided excellent waves for riding and playing.  Michael lost a couple of nose clips to the pounding surf.  We upgraded to the type with a neck strap which seemed to solve the problem.  Lisa and Sarah took advantage of the active waves and had some good runs with the boogie boards.

One hiccup for the week was a forgotten camera cable.  I carefully packed the charger but completely blanked on the computer cable.  We’ve been posting digital pictures during our travels for many years so that family and friends can keep up with our travels.  So, for this past week we had to largely go dark, just like the lighthouse.  During the week I had been reduced to posting a few pictures from my phone since I can download those.

Trying to avoid the heat of the day at the beach on Tuesday, we went into town to bowl (candle pin) and check out a few shops.  The bowling alley was very hot.  We only bowled one game.  We then explored a few of our favorite shops, including The Goldenrod where Lisa purchased some coconut clusters, an annual favorite!

This year we also tried a couple of loaves of bread from the “When Pigs Fly” bakery.  We have seen the store several times on Route 1 driving into York but had never stopped in.  However, when we went to do our grocery shopping we saw some of their loaves at the local Hannaford grocery store and purchased the “Six-Grain and Pumpkin Seed” loaf.  Sarah and I loved it, both buttered or with cheese.  Later during the week we got their Sourdough loaf to bring home.

Beyond daily beach bumming, we enjoyed our daily treat of Brown’s Ice Cream (typically Sherbet for Michael and Grape Nuts for the rest of us).  We also spent an evening in Kittery to visit Crate and Barrel as well as the Kittery Trading Post.

Wednesday we wandered around York Beach Village where we discovered a new restaurant, Woody’s Brick Oven Pizza.  As we walked by we all caught the delicious aroma of the fresh dough baking.  We decided we would have to try it out and we did on Friday afternoon.  The pizza was great.  Lisa and Michael stuck with plain cheese while Sarah and I added mushrooms, feta cheese and hot and sweet sausage.  We heartily recommend this place for pizza.

Friday night we attended Spamalot at the Ogunquit Playhouse.  The production was super!  The two leads, Charles Shaughnessy (Mr. Sheffield from TV’s “the Nanny”) and Rachel York were fantastic!   We enjoyed all of the fun music and silly Monty Pythonesque humor performed by stellar and lively cast, musicians and crew.

Present in the audience were several cast members for the next musical they are staging, Chicago.  Sally Struthers (Gloria from “All In The Family”), who is part of the Chicago cast, was there as well.  Although she has been in several Ogunquit productions, her appearances haven’t aligned with our schedule.  So we finally got to see her in person, though not on stage.

As usual, our week flew by, winding down as we met the remnants of Tropical Storm Earl later Friday night.  Although there were warnings posted at the beach to watch out for strong currents, we really didn’t experience more than a little wind and light rain.

By Saturday morning the clouds of Earl had mostly cleared out.  We fed the seagulls our week’s supply of left-overs and stopped by Stonewall Kitchen for some jams and assorted odds-and-ends.  We then headed south out of York and had a nice drive home, stopping by Fuddruckers for our traditional heading-home lunch.

From my point of view SQL injection is very well understood and has been for many years. There is no excuse for a programmer to create code that allows for such an attack to succeed. For me this issue falls squarely on the shoulders of people writing applications. If you do not understand the mechanics of SQL injection and don’t know how to effectively prevent it then you shouldn’t be writing software.

The mechanics of SQL injection are very simple. If input from outside an application is incorporated into a SQL statement as literal text, a potential SQL injection vulnerability is created. Specifically, if a parameter value is retrieved from user input and appended into a SQL statement which is then passed on to the RDBMS, the parameter’s value can be set by an attacker to alter the meaning of the original SQL statement.

Note that this attack is not difficult to engineer, complicated to execute or a risk only with web-based applications. There are tools to quickly locate and attack vulnerable applications. Also note that using encrypted channels (e.g. HTTPS) does nothing to prevent this attack. The issue is not related to encrypting the data in transit, rather, it is about keeping the untrusted data away from the backend RDMBS’ interpretation environment.

Here is a simple example of how SQL injection works. Assume we have an application that accepts a last name which will be used to search a database for contact information. The program takes the input, stores it in a variable called lastName, and creates a query:

String sql = "select * from contact_info where lname = '" + lastName + "'";

Now, if an attacker tries the input of: ‘ or 1=1 or ’2′=’

It will create a SQL statement of:

select * from contact_info where lname = '' or 1=1 or '2'=''

This is a legal SQL statement and will retrieve all the rows from the contact_info table. This might expose a lot of data or possibly crash the environment (a denial of service attack). In any case, using other SQL keywords, particularly UNION, the attacker can now explore the database, including other tables and schemas.

Sometime developers attempt to sanitize or “clean up” the incoming data. This is a valuable technique and should be a part of a defense-in-depth strategy. However, it is vital that the external input never be passed to the database in a way that the database engine could interpret it as part of the DDL or DML. Most modern development environments make this issue quite easy to prevent.

The best rule to establish when coding database interactions is to always use prepared statements. They are no harder to use than direct execution of the SQL and can be used to improve performance if the same query is being used more than once.

In any language that supports prepared statements the technique is always the same. You setup a template statement with placeholders for the dynamic (externally supplied) data. The dynamic portion will be filled-in using bound variables and will not be treated as part of the SQL statement itself. This means that there is no way a carefully crafted input can trick the environment and alter the structure of the SQL statement.

Here is an example of using a prepared statement in PHP:

$stmt = $dbh->prepare("select * from contact_info where name = ?");
if ($stmt->execute(array($_GET['lastName']))) {
  while ($row = $stmt->fetch()) {
  print_r($row);
}

Here is a prepared statement example in Java:

PreparedStatement prepSt =
      con.prepareStatement("select * from contact_info where name = ?");
prepSt.setString(1, lastName);
ResultSet rs = prepSt.executeQuery();
while (rs.next()){
  // Process the data
}

And here is a prepared statement example in C#:

using (SqlCommand myCommand =
    new SqlCommand("select * from contact_data where lname = @LNAME", myConnection)) {
  myCommand.Parameters.AddWithValue("@LNAME", lastName);
  myConnection.Open();
  SqlDataReader myReader = myCommand.ExecuteReader())
  // process data
}

What I hope you notice is that using prepared statements doesn’t add a lot of extra work to writing the program. That is why I don’t believe that there is any excuse for developers to be writing code that is vulnerable to such an attack. In all the examples provided (above) the input (in the variable lastName) will only be used to search the lname column, it will never be interpreted as part of the SQL syntax.

Next time you find yourself writing or maintaining code that connects to a database, please verify that you have done your due diligence by using prepared statements in all cases of database interaction. Doing anything less means that you are part of the problem rather than being part of the solution.

I know that there are other techniques, including the use of stored procedures, which are also used as a mitigation approach. Each technique has it place, but the use of prepared statements can typically be used regardless of other mitigating strategies. Again, it is all about defense-in-depth. Using a prepared statement is a simple yet powerful technique to create more secure software.

I am curious to hear from software authors that believe that they have to write software that is vulnerable to SQL injection. Is there some reason, other than a lack of understanding, that we as members of the software engineering profession continue to produce such vulnerable applications?

¹http://www.japantoday.com/category/crime/view/hackers-steal-customer-data-by-accessing-supermarket-database
²http://www.theregister.co.uk/2010/07/08/pirate_bay_hacked/
³http://www.computerworld.com/s/article/9177904/Mass_Web_attack_hits_Wall_Street_Journal_Jerusalem_Post?taxonomyId=17

My objectives for this first GUI were basic:

1. Support input of a set of triples in any format that Jena supports (e.g. REF/XML, N3, N-Triples and Turtle)
2. See the inferences that result for a set of assertions
3. Create a tree view of the ontology
4. Make it easy to use SPARQL queries with the model
5. Allow the resulting model to be written to a file, again using any format supported by Jena

Here are some screen shots of the application.  Explanations of the tabs are then provided.

The program provides each feature in a very basic way.  On the Assertions tab a text area is used for entering assertions.  The user may also load a text file containing assertions using the File|Open menu item.  Once the assertions are entered, a button is enabled that allows the reasoner to process the assertions.  The reasoner level is controlled by the user from a drop down.

Any inferences are shown in a separate text area on the Inferences tab.  The inferences are displayed using the same format as was used for the assertions.  A tree view of the model is also created and presented on a third tab called Tree View (creative, I know).

Along with presenting the inferences and tree view, the model may be queried using SPARQL.  The SPARQL tab contains a text area for the query, a button to run the query and a text area which displays the results.

The processed model may also be written out to a file.  By default it is written using the format used for the original assertions.  However, the format may be changed by selecting a specific language from the Setup menu.  The setup menu also allows the user to decide whether to only save the assertions or to save both the assertions and inferences.

A simple semantic model is provided with the download of the project, using Wikipedia’s Turtle example, as a way to try out the application.  Note that you need to have Java 5 and Ant installed in order to run the program easily.  I am hoping that BitRock InstallBuilder will approve my project for their open source license option.

There are a lot of opportunities to improve the operation of this tool.  Allowing external data sources as part of the SPARQL query is something I intend to add shortly.  The presentation of the SPARQL output should be in a table rather than a text area.  The tree view needs to be greatly expanded, allowing for exploration to arbitrary levels of depth, properly reflecting class hierarchy, allowing for exploration by subject, predicate and object as well as grouping the URIs (e.g. not showing multiple entries for the same URI under a given node).

Additional functionality, including features such as extracting data from relational databases, would also make sense to add.

As I mentioned in a previous post, I have created a SourceForge project, Semantic Workbench, with the intent of creating a tool that will contain a variety of features useful when working with semantic technologies, including the ones that I have been releasing on my website.  To that end, this small Java application may serve as a starting point for thinking through that tool.

If you have thoughts on tool features that would make working with semantic web concepts easier please comment here or better yet, join the SourceForge project.

In the meantime, if you are toying with semantic concepts or want to see how to leverage some of the features of Jena, feel free to download this application.

The program is really a proof-of-concept.  It takes a SQL query and converts the resulting rows into assertions of triples.  The approach is simple: given a SQL statement and a chosen primary key column (PK) to represent the instance for the exported data, assert triples with the primary key column value as the subject, the column names as the predicates and the non-PK column values as the objects.

Here is a brief sample taken from the documentation accompanying the code.

• Given a table named people with the following columns and rows:

       id    name    age
       --    ----    ---
       1     Fred    20
       2     Martha  25

• And a query of:  select id, name, age from people
• And the primary key column set to: id
• Then the asserted triples (shown using Turtle and skipping prefixes) will be:

       dsr:PK_1
          a       owl:Thing , dsr:RdbData ;
          rdfs:label "1" ;
          dsr:name "Fred" ;
          dsr:age "20" .

       dsr:PK_2
          a       owl:Thing , dsr:RdbData ;
          rdfs:label "2" ;
          dsr:name "Martha" ;
          dsr:age "25" .

You can see that the approach represents a quick way to convert the data.

The next question is, “How do I refactor the data?“  That was, after all, what my previous blog entry was discussing.  The decision for me becomes whether I need to add a bunch of features to the export program or is there a way to use features of the semantic web (e.g. OWL, SWRL) to refactor the data?

I compare this in some ways to the initial XML specification that required a DTD as the way to define the valid structure for an XML document.  The DTD is expressed using a different (not XML) meta-language.  This proved a poor choice, needlessly complicating parsers as well as developer’s learning curves.  The subsequent improvement was a move to XML schema, itself expressed using XML.  This added a consistency, using XML to describe XML.

I view the use of OWL (and probably something like SWRL if it continues to evolve) as a way to use a consistent technology to deal with data refactoring.  After all, if I am creating RDF data using semantic technologies and need to modify the data structure in some way (changing class or property names, adding classifications, etc.) then it makes sense to use the same semantic technologies to affect the transformation.

In the sample program I do just that.  I load an ontology file that contains my conversion assertions and then create the RDB-sources triples, allowing the reasoner to assert my changes.

I have released the program as open source.  The code and some documentation are available for download on my RDB To RDF web page.

At this point I’ve started small in terms of the inferencing, simply adding a superclass relationship and class membership based on a property value.  My goal was to get version one complete, creating a starting point on which to build out functionality.

Please feel free to download, use, and modify the program.  If you have feedback about its operation and the concepts being discussed please add a comment.

Both the business and IT are positioned to make-or-break the use of process automation tools and techniques. The business must redefine its processes and operational rules so that work may be automated.  IT must provide the infrastructure and expertise to leverage the tools of the process automation trade.

Starting with the business there must be clearly defined processes by which work gets done.  Each process must be documented, including the points where decisions are made.  The rules for those decisions must then be documented.  Repetitive, low-value and low-risk decisions are immediate candidates for automation.

A key value point that must be reached in order to extract sustainable and meaningful value from process automation is measured in Straight Through Processing (STP).  STP requires that work arrive from a third-party and be automatically processed; returning a final decision and necessary output (letter, claim payment, etc.) without a person being involved in handling the work.

Most businesses begin using process automation tools without achieving any significant STP rate.  This is fine as a starting point so long as the business reviews the manual work, identifies groupings of work, focuses on the largest groupings (large may be based on manual effort, cost or simple volume) and looks to automate the decisions surrounding that group of work.  As STP is achieved for some work, the review process continues as more and more types of work are targeted for automation.

The end goal of process automation is to have people involved in truly exceptional, high-value, high-risk, business decisions.  The business benefits by having people attend to items that truly matter rather than dealing with a large amount background noise that lowers productivity, morale and client satisfaction.

All of this is great in theory but requires an information technology infrastructure that can meet these business objectives.

On the IT side we must have an information infrastructure that promotes automation.  Key components for process automation are workflow and rules engines and the expertise to use them effectively.  This is a major undertaking on its own.

However, workflow and rules engines are simply software components that provide features such as Service Level Agreement (SLA) management, long-running work state management, independently codified rules, rule versioning and so forth.  These engines need access to data and that data must be structured and clearly defined.

Therefore, in order to create an IT infrastructure ready to leverage process automation there must be a set of well-defined data services that all applications (manual and workflow-based) use to read and write data. Those services, which may be web services or message-based solutions, become the common approach by which all applications interact with the data sources.

By utilizing this centralized service approach, applications that allow for manual manipulation of data, through some form of user interface, can be leveraged by a workflow or rules engine to carry out the work on behalf of a person.  The service doesn’t know or care if it is being invoked by a person or system, it simply reads and writes the proper data.

In the article I also discuss the advantages of leveraging process automation beyond simply speeding up processes.  The platform simplifies the inclusion of additional channels (IVRs, portals) and third party access (vendors, clients) to your systems.

The full article can be found on InformationWeek Analytic’s subscription-based website at http://analytics.informationweek.com/abstract/22/3593/SOA-App-Architecture/strategy-process-automation.html.

If you are interested in reading the article but are not an InformationWeek Analytics subscriber, there are a limited number of copies that Blue Slate can provide.  Just drop me a note at david.read@blueslate.net and I’ll follow-up with you.

As always, I am interested in hearing about the successes and lessons-learned from people that are working with process automation.  So please share your insights and stories by adding your comments.

After parking we rushed past the Food Circle in order to get to the North Yarmouth Academy Gymnasium and receive our instructions and umbrella hats for “Maine’s Longest Smile” being organized by The Levity Project.  Although temperatures as we drove through Massachussets and New Hampshire and up into Maine were in the 90s, by the time we got to Yarmouth it was about 75.  The gym was another story, hot and humid, but full of festivity!

Hippity-hop balls were being test driven by a variety of people while others were testing out the new hats.  We signed in, filled out a photo release and starting reviewing the instructions we had been given.  By 5pm we were being walked through the overall plan and were ready to head out to our assigned locations by 5:30.  Before getting into position I snuck by the First Parish Congregation Church for my annual lobster roll.  They make a perfect lobster roll – a bun and lots of lobster meat!  Nothing else to distract from the delicious lobster flavor.

For the smile project our family was assigned to group three, which had a great location right by the Memorial Green tent and Food Circle.  The street was lined with people prepared to watch the parade.  The levity event went off without a hitch and we had a blast!  Participating in the smile project was fun.  It was a chance to be a part of the festival and not just an out-of-town visitor watching everything from the sidelines.

Once the smile wrapped-up we headed over to grab some dinner (my lobster roll having been a perfect appetizer).  Sarah had calamari, Michael went for clam strips, Lisa found some chicken and I ordered the whole clams fried with crumbs (saving the batter-dipped for Saturday).

We enjoyed our meal and went over to explore the craft booths.  The variety and quality of the work by the crafters makes the exploration fun.

For the first time we skipped the rides.  Neither child was interested in heading into the carnival – perhaps the length and heat of the day had tired them out.  We headed for a nearby hotel and relaxed.

Saturday we returned to the festival and explored the art show tents, which close early each night so we missed them Friday.  We then had lunch. I had a Lime Rickey and fried clams in batter which were perfect!

Later in the afternoon we headed up to Belgrade, Maine where Lisa’s sister and brother-in-law have a lake-side camp.  We celebrated our nephew Sam’s 20th birthday and enjoyed swimming and good company.  I think that Michael, ever the water bug, spent more time in the water Saturday and Sunday than the rest of us combined!

Sunday afternoon we headed back, stopping by York Beach for some ice cream at Brown’s Ice Cream (Grape Nut as usual for Lisa, Sarah and me; Raspberry Sherbet for Michael).  Dinner followed later at Fuddruckers in Massachusetts.  Unfortunately I couldn’t convince Lisa to try the buffalo burger (we always split a burger since they are quite large).  By 10:30pm we were home and getting organized for a new week.

As always we enjoyed the trip and look forward to our vacation in York Beach later in the summer.

Thank you to Leslie Wagner (http://www.lesliewagnerphoto.com/) for allowing me to share her photos from The Levity Project event.

I started pursuing the certification in mid-2009, got serious about studying early this year (2010), took the exam in late April, was notified that I passed and had my background endorsed in May, had to update my resume for an auditor in early June and was awarded the CISSP designation at the end of June.

I felt that this certification was important both professionally and personally.

Professionally, the certification serves as a validation that I have a solid and broad understanding of information systems’ security.  People who have worked with me know that I have been focused on IS security for many years.

Whether performing security-centered code reviews, fixing flawed implementations or teaching designers and developers how to improve the security of their systems, I have been on a mission to mentor and train people to observe effective security practices and principles.  I’ve also had operational responsibility for system infrastructures.  With that experience I was able to pass GIAC’s GSEC and Red Hat’s RHCE exams several years ago.

Personally, the process of studying and passing the exam allowed me to pursue and attain a non-trivial goal.  I am enrolled and taking classes toward my master’s degree, but completing that work will require several more years of part time attendance.  Setting and achieving intermediate goals helps to keep me focused and learning.

If you are wondering what the CISSP is all about, please read on.

The (ISC)²‘s CISSP Common Body of Knowledge (CBK) covers 10 security-centric domains. It is this CBK that forms the basis of the CISSP exam.  The domains cover a broad array of subject areas including physical security, operations security, cryptography, business continuity, network security and so forth.

The bulk of my experience falls solidly into three of the domains: Application Security; Cryptography; and Telecommunications and Network Security.  I have had operational experience in several other domains and there are a few which represent areas well outside any of my professional responsibilities.

I began studying in 2009 by reading the CISSP for Dummies book^b.  I wanted something that would give me a quick overview of the domains and exam process.  The book was helpful, though insufficient as my sole source for understanding the breadth and depth of the CBK.  One suggestion the book made, which later turned out to be quite wise, was to sign up for the exam in order to force oneself to buckle down and study.

I next read the (ISC)²‘s official guide to the CISSP CBK^c.  Although somewhat dry, the book gave me a solid understanding of the knowledge expectations that the (ISC)² has for someone to pass the exam and represent the profession.  I took a long time to finish this book, putting it aside at times to read books on other subjects.

In February of 2010 I finally took the advice from the “For Dummies” book and registered for the exam that was scheduled to take place nearby in April. Now the clock was truly ticking and I wanted to assess where I stood.  I knew that I was solid in some domains but likely weak in others.

In order to have access to a broad set of practice exam questions, I purchased Shon Harris’ CISSP exam guide^d.  This book is well written and very approachable.  There are lighthearted comments sprinkled throughout the book which help to make the material more engaging.  The book also supplies a testing tool with a nice set of practice questions.

After completing the Harris book I took a practice exam and identified my weakest domains (Security Architecture and Design, Physical Security).  I applied some focused study around these 2 areas and took another practice exam, passing with better than 95% across all domains.  At that point I felt I was ready, which was good since it was getting close to exam day.

A couple of weeks before the exam I received an email containing my exam admission document.  It spelled out the location and process for taking the exam.  This information agreed with the background provided in the books I had read.  Essentially you don’t need anything beyond the admission document and an acceptable photo id (such as a driver’s license).

You can usually bring something to drink and eat, since you may be there for up to 6 hours.  Where I took the exam, all the test takers were required to put their drinks and snacks at the back of the room and go there to partake.

While taking the test, time flew by for me.  I had no idea what time it was when I left since I didn’t have my cell phone with me (electronic devices were not allowed in the exam room).  It turned out that I had taken about 3 hours to finish.

Over the next few months I intend to blog about each domain including highlights around material that was new to me as well as information that I already knew and was clearly germane to the CBK.

I am interested in hearing about experiences from others that have taken the exam or are contemplating taking it.  Since I have a background in education I’m looking for an opportunity to assist with a study group for prospective CISSP test takers.  Perhaps we can get a group started in the Albany/Schenectady area.

^a Website: ISC^2
b Book: CISSP for Dummies
^c Book: Official (ISC)² Guide to the CISSP CBK
^d Book: CISSP All-in-One Exam Guide

The basis for the program is an example program that is described in Hebler, Fischer et al’s book “Semantic Web Programming” (ISBN: 047041801X).  The intent of the program is to load an ontology into three models, each running a different level of reasoner (RDF, RDFS and OWL) and output the resulting assertions (triples).

I made a couple of changes to the book’s sample’s approach.  First I allow any supported input file format to be automatically loaded (you don’t have to tell the program what format is being used).  Second, I report the actual differences between the models rather than just showing all the resulting triples.

As I worked on the code, which is currently housed in one uber-class (that’ll have to be refactored!), I realized that there will be lots of reusable “plumbing” code that comes with this type of work.  Setting up models with various reasoners, loading ontologies, reporting triples, interfacing to triple stores, and so on will become nuisance code to write.

Libraries like Jena help, but they abstract at a low level.  I want a semantic workbench that makes playing with the various libraries and frameworks easy.  To that end I’ve created a Sourceforge project called “Semantic Workbench“.

I intend for the Semantic Workbench to provide a GUI environment for manipulating semantic web technologies. Developers and power users would be able to use such a tool to test ontologies, try various reasoners and validate queries.  Developers could use the workbench’s source code to understand how to utilize frameworks like Jena or reasoner APIs like that of Pellet.

I invite other interested people to join the Sourceforge project. The project’s URL is: http://semanticwb.sourceforge.net/

On the data side, in order to have a rich semantic test data set to utilize, I’ve started an ontology that I hope to grow into an interesting example.  I’m using the insurance industry as its basis.  The rules around insurance and the variety of concepts should provide a rich set of classes, attributes and relationships for modeling.  My first version of this example ontology is included with the sample program.

Finally, I’ve added a semantic web section to my website where I’ll maintain links to useful information I find as well as sample code or files that I think might be of interest to other developers.  I’ve placed the sample program and ontology described earlier in this post on that page along with links to a variety of resources.

My site’s semantic web page’s URL is: http://monead.com/semantic/
The URL for the page describing the sample program is: http://monead.com/semantic/proj_diffinferencing.html

This quote has been running through my head a lot as I’ve been spending time alone in church.  Our church music director will be away for the first three Sundays in June and asked if I would be willing to take the reins during his absence.  This isn’t the first time I done this, but I believe it is the longest stint I’ve had.

My initial thinking, and usual approach, when playing the service is to use the piano to accompany everything.  This is where I feel safest.  I spend a lot of time at the piano, working with the Children’s Choir, rehearsing with the Brass Kickers and accompanying myself for a variety of solos and duets.

However, I’ve begun to feel compelled to use the organ.  Its variety of colors, range of tones and wide dynamic range cannot be approached by the piano.  Although I love the sound of a piano accompanying a solo voice, the organ adds significant sonic breadth, especially when accompanying hymns.

In fact, I believe that it is the flatness of verse after verse of a hymn played on the piano that has continually pushed me to move out of my personal comfort zone and explore the organ as a more versatile and ultimately more appropriate instrument for such situations.  To be sure, it is now taking me an inordinate amount of time to prepare for a service.

When using the piano, all I needed to do was learn to play the notes.  Now I need to worry about the voicings for each verse.  Looking at the text to suggest color and dynamics adds work.  Basic tasks such as figuring out which manual to use for each verse and configuring piston settings so that they are convenient to access while playing also adds complexity for someone that does not use the instrument often.

I have great respect for those that make such planning and preparation look easy.  I cannot imagine doing this week after week, at least not with a separate full time job.  I would guess that over time one would get to know the instrument and have a more organized approach to this process.  For me there is a great deal of experimentation, figuring out which ranks extend into which octaves and which timbres sound well together.

To be sure, it is an amazing experience to fiddle with such decisions and hear the difference in the feeling invoked by a given hymn.  Played with certain stops, the piece is upbeat.  Change the sounds and it is suddenly reflective or pensive.  In fact, my biggest risk is probably over-using the breadth of sounds and dynamics.

For instance the carillon seems like a great choice for bringing out a melody, perhaps to introduce a hymn.  However, it would probably be tiresome for the congregation if used every week.  Also, it is tempting to approach some hymns with a powerful accompaniment.  I enjoy hearing the reverberation at the end of the piece while practicing.  However, the congregation shouldn’t be in a shouting match with the organ, so I’ll need to tame my “Virgil Fox-ness”¹.

This will be in interesting month of Sundays for me and the congregation.  I pray we will each find enjoyment and meaning during the worship time spent together.

At the least, I hope those worshiping don’t come away saying, “I like the silent church before the service begins, better than Dave’s organ playing.”

¹http://en.wikipedia.org/wiki/Virgil_Fox

The refactoring of a database becomes less and less trivial as project development continues.  While developers have IDE’s to refactor code, change packages, and alter build targets, there are few tools for refactoring databases.

My definition of a database refactoring tool is one that assists the database developer by remembering the database transformation steps and storing them as part of the project – e.g. part of the build process.  This includes both the schema changes and data transformations.  Remember that the entire team will need to reproduce these steps on local copies of the database.  It must be as easy to incorporate a peer’s database schema changes, without losing data, as it is to incorporate the code changes.

These same data-centric complexities exist in waterfall approaches when going from one version to the next.  Whenever the database structure needs to change, a path to migrate the data has to be defined.  That transformation definition must become part of the project’s artifacts so that the data migration for the new version is supported as the program moves between environments (test, QA, load test, integrated test, and production).  Also, the database transformation steps must be automated and reversible!

That last point, the ability to rollback, is a key part of any rollout plan.  We must be able to back out changes.  It may be that the approach to a rollback is to create a full database backup before implementing the update, but that assumption must be documented and vetted (e.g. the approach of a full backup to support the rollback strategy may not be reasonable in all cases).

This database refactoring issue becomes very tricky when dealing with multiple versions of an application.  The transformation of the database schema and data must be done in a defined order.  As more and more data is stored, the process consumes more storage and processing resources.  This is the ETL side-effect of any system upgrade.  Its impact is simply felt more often (e.g. potentially during each iteration) in an agile project.

As part of exploring semantic technology, I am interested in contrasting this to a database that consists of RDF triples.  The semantic relationships of data do not change as often (if at all) as the relational constructs.  Many times we refactor a relational database as we discover concepts that require one-to-many or many-to-many relationships.

Is an RDF triple-based database easier to refactor than a relational database?  Is there something about the use of RDF triples that reduces the likelihood of a multiplicity change leading to a structural change in the data?  If so, using RDF as the data format could be a technique that simplifies the development of applications.  For now, let’s take a high-level look at a refactoring use case.

Imagine we are in the first iteration of a web-based on-line store and we decide to support only ordering one item.  We opt to store that item in a table with the order header data.  In the next iteration we decide to add support for multiple items in the shopping basket.

We’ll refactor the database to support a one-to-many relationship between the order header and the shopping basket items.  There is nothing wrong with this approach; it is simply a part of refactoring the design.  Beyond altering the database schema (adding the order_item table and removing columns from the order table), this change will necessitate a transformation of the existing order data into the two-table structure.

What is interesting to me is to look at this situation where the data is modeled using RDF triples.  What data structure, if any, changes between these two iterations?

The semantic relationships don’t change.  What does change?  Very little, depending on the implementation.  If the first iteration’s RDF triple for the order was an order (subject), itemOrdered (predicate) and item (object), then the data structure is unchanged for iteration 2.  We will simply have to allow multiple itemOrdered predicates on the order instance.

Based on this conjecture we are off to a promising start in terms of simplifying data refactoring.  In this case no transformation work was needed to the data itself.  If many of our refactoring use cases look like this (changes to the restrictions and not the relationships) then the use of RDF triples as the data storage format offers an attractive alternative to relational databases when dealing with changing data needs.

Of course that concept (changing data needs) is a key aspect of the semantic web.  The ability to add new data and structures without breaking old ones is a requirement for an extensible and decentralized web database.  It makes sense that looking at a use case within a smaller scope would expose the same benefit.

Will database refactoring of RDF triple-based structures always be this simple?  I don’t believe so.  If the semantic relationship changes, due to our learning more about the domain, then there will be actual structural refactoring needed.  If we start out using a datatype property for an object and then convert to an object property, we will have to restructure any existing instances.

To defend against these types of issues we need to focus on the correct ontology for our domain first, before building applications.  Any application then built for that domain (company, industry) will benefit, regardless of the development process (e.g. agile, waterfall).

Have you worked with RDF triple-based data structures as part of an agile project?  If so, do you have thoughts on whether the use of triples simplified data storage refactoring from iteration to iteration?

I’ll be trying these techniques on a POC and hope to have more concrete examples of the impact of this alternate data storage approach.  It is just another example of where semantic technologies are positioned to significantly impact the ways that we design, develop and test software-based solutions.