So why the posting?  Well, either my math skills are getting really poor (probably due to my relentless exposure to computers and calculators) or Target has some issue with understanding egg pricing.  Here are pictures of the price labels in the dairy section (taken at the Target in Clifton Park, NY on September 3, 2012).  Anything look askew?

Now, if the errors were consistent I guess I could understand.  After all converting from “12 eggs” to “price per dozen” takes some understanding of the word “dozen” along with the principle of unit pricing.  What I find interesting is that the unit pricing calculation for these eggs is somewhat cracked since it does not seem to follow a pattern.  Also, apparently no one has noticed these strange unit prices.

Unit pricing is supposed to help people make informed decisions about the price difference between options.  It is irritating enough when the unit price for a “value” or “saver” size is higher than the regular size.  But at least the shopper who is paying attention can readily identify the deception.

More troublesome is looking at a group of products and finding that their unit prices don’t use a consistent unit.  For example, I’ve looked at cereals and had some brands give a unit price per ounce and others per pound.  Worse, I’ve seen volume and weight measures used on similar products.  Clearly such a store doesn’t want me (the consumer) to have an easy path to understanding the relative prices of those items.

In this case the unit price is just plain wrong.  It cannot be used in any way to compare the relative prices of the eggs since the error isn’t consistent, creating some sort of shell game.  In several cases the unit price is correct, in others it is off by 12 (treating the price for 12 eggs as if it were the price for 1).  In one case the error is a factor of 8.  Where did they come up with 8?

Being both amused and frustrated by these errors, I’ll be taking a closer look at unit pricing at a variety of stores just to understand the reliability of these “consumer aids.” It will be interesting to compare the results between retail chains as well.

Anyone out there ever use unit price information when choosing which product to buy?  Ever found errors or confusing unit pricing schemes being used?

In my last post I talked about the fact that integration between components has to be based on the meaning of the data, not simply passing compatible data types between systems.  Semantic technology supports this need through its design.  What key capabilities does semantic technology offer in support of integration?  Here I’ll highlight a few.

Logical and Physical Structures are (largely) Separate

Semantic technology reduces the tie between the logical and physical structures of the data versus a relational database.  In a relational database it is the physical structure (columns and tables) along with the foreign keys that maintain the relationships in the data.  Just think back to relational database design class, in a normalized database all of the column values are related to the table’s key.

This tight tie between data relationships (logical) and data structure (physical) imposes a steep cost if a different set of logical data relationships is desired.  Traditionally, we create data marts and data warehouses to allow us to represent multiple logical data relationships.  These are copies of the data with differing physical structures and foreign key relationships.  We may need these new structures to allow us to report differently on our data or to integrate with different systems which need the altered logical representations.

With semantic data we can take a physical representation of the data (our triples) and apply different logical representations in the form of ontologies.  To be fair, the physical structure (subject->predicate->object) forces certain constrains on the ontology but a logical transformation is far simpler than a physical one even with such constraints.

Meaning is Inherent in the Infrastructure

Semantic technology includes a powerful ontological representation at its core.  This key capability, constraining terms as a way to create precise definitions, allows us to represent information with clear and unambiguously defined meaning.  More to the point, by starting with precise definitions at an enterprise level, we can assure that each system ties to that ontology through related system-level ontologies.

The “meaning” aspect of semantic technology is valuable in the Cognitive Corporation™ since the data from all systems is being integrated and used to inform our predictive models and ultimately our applications, through workflow, rules and data changes.  Having a concise set of definitions that each system can tie to allows us to incorporate legacy systems as well as create new systems each sharing the concepts from our enterprise ontology.

The integration of operational data and behavioral data is vital to a learning organization. In order to learn it is not good enough to see that some value is changing, we have to be able to track down why the value is changing.  This means that the operational data alone is insufficient to inform learning. Operational data allows us to build models representing what is known to be happening. It is not particularly useful in helping to identify the root cause.  Our broader behavioral data is required and must be correlated with the operational data.  Semantic technology supports this requirement very well.

Flexibility in Data Representation through URIs

When integrating systems we may need to work with a variety of data, including rich data such as documents and media files.  The structure used to represent data in semantic solutions gives us significant latitude in terms of the data being represented.  We can create relationships to constants (string, integer, floating point and so forth) but of more interest is how we create relationships to other objects.  That is done through URIs.

This simple approach to data representation, using URIs, removes a great deal of noise from the design of our database.  It also allows us to take advantage of an ongoing trend to move unstructured information into Content Management Systems (CMS).  Any modern CMS will permit referencing its information through a URL – easily integrated with our semantic data store.  Further, integration between systems, even those from other businesses, is much more reasonably accomplished through URLs than copying data or creating proprietary integrations.

Another benefit to representing objects through the use of URLs is the fact that the content represented by a given URL can differ – an image in one case, a video in another and an MP3 in a third.  We do not need to bother our data tier with these details.  When representing data we are interested in intent, the logical relationships.  Rendering is not the job of the data tier – so why add that overhead to our data infrastructure?  It is the data tier’s job to enable the logical structure so that we can find what we are looking for.  What to do with the data once we find it is the job of other tiers in our solution stack.

I hope this post shines a little light on how semantics plays a key role in supporting a Cognitive Corporation™.  As mentioned in an earlier post, we will be putting these concepts to the test with a set of enterprise products.  The team looks forward to being able to share their insights and lessons learned.

Are you looking at semantic technology within the scope of your enterprise architecture?  What do you see as its principal benefits, if any?

I’ve blogged a couple of times (see references at the end of this blog entry) about the Cognitive Corporation™.  The depiction has changed slightly but the fundamentals of the framework are unchanged.

The focus is to create a learning enterprise, where the learning is built into the system integrations and interactions. Enterprises have been investing in these individual components for several years; however they have not truly been integrating them in a way to promote learning.

By “integrating” I mean allowing the system to understand the meaning of the data being passed between them.  Creating a screen in a workflow (BPM) system that presents data from a database to a user is not “integration” in my opinion.  It is simply passing data around.  This prevents the enterprise ecosystem (all the components) from working together and collectively learning.

I liken such connections to my taking a hand-written note in a foreign language, which I don’t understand, and typing the text into an email for someone who does understand the original language.  Sure, the recipient can read it, but I, representing the workflow tool passing the information from database (note) to screen (email) in this case, have no idea what the data means and cannot possibly participate in learning from it.  Integration requires understanding.  Understanding requires defined and agreed-upon semantics.

This is just one of the Cognitive Corporation™ concepts that we will be exploring in the lab environment.  We will also be looking at the value of these technologies within different horizontal and vertical domains.  Given our expertise in healthcare, finance and insurance, our team is well positioned to use the lab to explore the use of learning BPM in many contexts.

From an implementation perspective we have a core execution team of 8 individuals assigned as owners across Requirements, BPM, BRM, Data Mining, Semantic Technology, Relational Database Technology, and Application Architecture.  Scrum will be used as the lifecycle for the lab projects.  Beyond the deep expertise provided by our team, I’m excited by the energy and excitement that the execution team members bring to the lab.

A separate advisory team has also been established.  This group will provide a sounding board as the sprints complete and new stories are considered.  It will also serve as a pool of team members to swap into the execution team since these folks will be in the loop with the ongoing projects.

The tools we will use are enterprise solution components including workflow and rules engines, web service and message-based middleware, traditional relational database management systems, data mining tools and semantic technology such as an ontology editor, reasoner, triple store and SPARQL endpoint.  We are creating an architecture with components similar to what we find our clients using, augmented with less common components such as predictive modeling tools and semantics.

The lab’s goal is to advance our company’s expertise with these enterprise tools and techniques, providing insight regarding best practices for leveraging them as components of a learning system.  Methods for dynamic interactions between components such as predictive analytics driving business rules changes or semantic technology simplifying data integration will be tried, honed and ultimately become part of Blue Slate Solution’s best practices for advancing the value of BPM within our clients’ enterprises.

As part of our process, we will be sharing updates, both from individual team members through blog entries and also in general communications highlighting the overall progress and outputs from the lab.  We all hope that others will find the information thought provoking and educational.

If you have thoughts about what we are undertaking, or would like to learn more about it, please feel free to send an email to info@blueslate.net.

Prior blog entries on the Cognitive Corporation™:

• The Cognitive Corporation – An Introduction
• The Cognitive Corporation – Effective BPM Requires Data Analytics

What exactly is the issue?  Well, it turns out that the two letter postal abbreviations (for example AK for Alaska and HI for Hawaii) can’t be used as the key for sorting the state names into alphabetical order.  For the most part it works, however for some states, such as Nevada through New Mexico it breaks.  As a New York resident I get tripped up by this.

The image shown here is a web form for a college admissions site.  As you can see, Nevada follows New Hampshire, New Jersey and New Mexico but precedes New York.  In reality it should follow Nebraska and precede New Hampshire.  This order is incorrect; it is based on the state abbreviations.  If instead of state names the website were displaying the state abbreviations, the order would be NH, NJ, NM, NV, NY and all would be fine.

The developer(s) of this site are not alone in their mistaken use of the postal abbreviation as the sort key.  I’ve encountered this issue with online shopping sites, reservation systems and survey forms.  I typically do a quick “view source” of the site and invariably they are using the state abbreviation as the actual value being passed to the server.  I’m sure they are using that for sorting as well.

You might think this sort of thing doesn’t matter.  From my point of view it represents a “broken window,” using Andy Hunt’s and Dave Thomas’ language from The Pragmatic Programmer.  Little things count.  Little things left uncorrected form an environment where developers may become more and more sloppy.  After all, if I don’t need to pay attention to my sort key for state, what’s to say I won’t make a similar mistake with country or a product list or any other collection of values that is supposed to be ordered to make access easier?

Please, if you are designing an input form, make sure that sorted information displayed by your widgets is sorted by the display value, not some internal code.  It will make the use of your form easier for users and garner the respect of your fellow developers.

Have you seen this flaw on websites you’ve visited?  Do you have pet peeves with online form designs?  I’d enjoy hearing about them.

To begin the conference I attended the half day “Ontology 101” presented by Elisa Kendall and Deborah McGuinness.  They indicated that this presentation has been given at each semantic technology conference and the interest is still strong.  The implication being that new people continue to want to understand this art.

Their material was very useful and if you are someone looking to get a grounding in ontologies (what are they?  how do you go about creating them?) I recommend attending this session the next time it is offered.  Both leaders clearly have deep experience and expertise in this field.  Also, the discussion was not tied to a technology (e.g. RDF) so it was applicable regardless of underlying implementation details.

I wrapped up the first day with Richard Ordowich who discussed the process of reverse engineering semantics (meaning) from legacy data.  The goal of such projects being to achieve a data harmonization of information across the enterprise.

A point he stressed was that a business really needs to be ready to start such a journey.  This type of work is very hard and very time consuming.  It requires an enterprise wide discipline.  He suggests that before working with a company on such an initiative one should ask for examples of prior enterprise program success (e.g. something like BPM, SDLC).

Fundamentally, a project that seeks to harmonize the meaning of data across an enterprise requires organization readiness to go beyond project execution.  The enterprise must put effective governance in place to operate and maintain the resulting ontologies, taxonomies and metadata.

The full conference kicked off the following day.  One aspect that jumped out for me was that a lot of the presentations dealt with government-related projects.  This could have been a side-effect of the conference being held in Washington, DC but I think it is more indicative that spending in this technology is more heavily weighted to public rather than private industry.

Being government-centric I found any claims of “value” suspect.  A project can be valuable, or show value, without being cost effective.  Commercial businesses have gone bankrupt even though they delivered value to their customers.  More exposure of positive-ROI commercial projects will be important to help accelerate the adoption of these technologies.

Other than the financial aspect, the presentations were incredibly valuable in terms of presenting lessons learned, best practices and in-depth tool discussions.  I’ll highlight a few of the sessions and key thoughts that I believe will assist as we continue to apply semantic technology to business system challenges.

Marcel Jemio from the Department of the Treasury discussed a massive project seeking to apply semantics to their siloed transactional data.  His focus was to present his lessons learned as well as a possible blue print for similar projects.  He did a great job on both fronts.  A focus he suggested was that such projects should not be concerned with classifying data; rather they should seek to classify knowledge.  It is knowledge that is actionable and provides meaning and relevance.

Some other important points he made included the need for business data governance, standardized data (vocabulary, semantics) and a shared information repository.  Beyond his excellent sharing of experiences, his slides included recommendations for how to present information to business stakeholders so as to inform and educate.

Richard Green’s presentation looked at how to go about creating a practical ontology for an enterprise.  He dove into detail, showing some of the structured approaches that he uses to gather the requirements that feed an ontology design.  A key point for Richard was that ontologies must be practical (delivering value, being easy to use), define an ontology (describing what exists and their relationships) and work for a specific enterprise or constituent.

I thought his approach was very logical and included useful guidance for thinking through the process.  Sharing his documentation templates helped to clarify how the approach works and specifically how his “5 w’s” are applied to different situations.  I intend to integrate some of his approach into my practices.

Another presentation that contained specific guidance was Janet Millenson’s ROI of RDF discussion.  She outlined some of the key drivers that would lead an organization to apply RDF-based technology as a more efficient way to approach a business challenge.  She went beyond simple ROI and also discussed the underlying business driver such as reducing costs and supporting their mission.

Her approach to making the business case was standard procedure for introducing large scale change to an organization’s leadership.  Like Marcel, she cautioned against the use of technology terms in these types of discussions.  The technology is irrelevant to business leadership, the results are what matter.  Based on her experience she shared the importance of setting realistic expectations and creating the right team in order to succeed.

When discussing actual project planning and execution she gave a good overview of some major aspects that must be thought-through and managed in order to succeed.  These include people with their skills as well as technologies leveraged and the business domain being addressed.  Considering technology issues (data quality, governance), organizational concerns (build/buy, vendor relationships) and market risk (vendor consolidation, standards evolution) are vital in order to create the most effective result.

Her bottom line was that a successful project will address a specific problem in the organization (not boil the ocean), understand and work within constraints, start with realistic expectations and plan flexibility into the result to deal with technology and business changes.

David Booth (presenting on behalf of Jurgen Angele) walked through 2 healthcare industry case studies where semantic technology was used to effectively advance the effectiveness and safety of care while reducing costs.  The initial case was from the Cleveland Clinic.

The usual starting point of having multiple stovepipe systems that contained related data was presented.  The total data set, if combined in a meaningful way, would create a much more useful information source for doctors and researchers. Since semantic technology simplifies the integration of separate data sources, it was an obvious choice for this project.

David described the high-level process that was used to bring the data together.  The basic subsumption capability of inferencing is a simple yet powerful way to quickly combine data.  He quoted Jim Hendler, “A little inferencing goes a long way.”  The presentation described the pipeline process that was applied and which makes a lot of sense as a standard approach to dealing with such integration projects.

The second case study revolved around PanGenX, which seeks to use human genome information to understand which drugs will work for which individuals.  The overall data integration process was very similar to the first case study.  One point that was made, which resonated with me, was the statement that SPARQL is a convenient rule language.  This comment was in the context of simple rules, obviously not inferencing.  However if simple rules are being created, having them in the same syntax as the queries simplifies the process.

Neil Raden’s presentation, albeit abbreviated, was thought provoking.  Neil’s direct manner, devoid of pretense, allowed him to actually cover a lot of information in a very short time.  His take on the overhyped and overly complex processes that represent current BI practices, such as large MDM projects and monolithic data warehouses, was that these are not where businesses should be spending their BI dollars.

He dropped a lot of information in one-liners.  Some of which weren’t related to semantics as much as data in general.  I found his presentation a call to action to leverage semantic technology in support of a new data paradigm, not a new way to implement existing paradigms.  I don’t know if that was a point he intended to make but it seemed like a logical extension of what he was saying.

The conference wrapped up with a panel discussion between David Booth, Elisa Kendall, David McComb and Dennis Wisnosky.  They discussed some of the changes they have seen with semantic technology including the fact that project implementation seems to be getting easier as tools mature and the semantic technology ecosystem gets fleshed out.  They are also seeing people focusing on more pragmatic and meaningful projects, going beyond simply playing with the technology.

There was some discussion about the continuing hype associated with this area and the need to not fall prey to that.  The work that has been done represents hard work and collegiate sharing.  The solutions are neither turnkey nor ready to simply buy and install.

There continues to be a focus on unstructured data which is good since so much corporate information is sitting idle in documents rather than databases.

Some specific guidance for getting started with semantic technology: prove it to yourself with a POC.  You can’t discuss it more broadly if you don’t believe in it.  Choose an initiative and give it a name so that people recognize it.  Plan to deliver value on a regular (no less than every 6 months) basis.  Start small and grow from there.

Controlling scope, as in all projects, is central to working with new technology in order to get to a success point and begin acquiring lessons learned, which will allow for future success.  The guidance from “The Mythical Man Month” is still relevant: you always throw the first one away.

A theme that was heard in several sessions and repeated in the panel discussion was that use cases must be leveraged when beginning a semantic technology project.  In reality that has nothing to do with semantics.  I leverage use cases (or user stories) in every business system project.  The fact that they are required for semantic technology projects simply points out that these projects are still focused on business success and must be driven by business needs.

On the technology side there were several vendors present demonstrating their tools and providing sessions that explored their case studies.   The capabilities of these tools continue to advance quickly.  I saw several that I will be trying out including ontology editors, inference engines and integration tools.

It is great to see the tool landscape continuing to grow in breadth and depth.  It is also nice to see a mix of commercial and open source alternatives, allowing a company to start a small POC without a large capital investment and then having commercial alternatives based on needs.

Overall the conference presented useful experiential information through case studies and lessons learned.  More commercial experience needs to be shared.  Also, a way needs to be found to make the output and advancement that the working groups are driving more visible and understandable.

In my case I am not looking to build a triple store or an OWL interpreter, those types of standards are not interesting to me.  I need more basic information.  Some of this exists but may not be as visible as it could be.  There was discussion around the importance of raising the visibility of things like CIO-level information and developer (of semantic technology-based applications, not the semantic technology itself) documentation.

I continue to be convinced that semantic technology will fundamentally change the way we work with data and system integrations.  It is simply a matter of time to get there.  Practitioners and thought leaders have an important role to play in order to get this vision out into the mainstream and keep it in the forefront of enterprise system conversations.

As always, if you have thoughts you’d like to share about a topic on my blog, feel free to add a comment.

On the Android platform it is actually quite easy to execute SPARQL against remote SPARQL endpoints, RDF data and local models.  The heavy lifting is handled by Androjena’s ARQoid, an Android-centric port of HP’s Jena ARQ engine.

Both engines (the original and the port) do a great job of simplifying the execution of SPARQL queries and consumption of the resulting data.  In this post I’ll go through a simple example of using ARQoid.  Note that all the code being shown here is available for download.  This post is based specifically on the queryRemoteSparqlEndpoint() method in the com.monead.androjena.demo.arqoid.SparqlExamples class.

Setup

To begin, some environment setup needs to be done in order to have a properly configured Android project ready to use ARQoid.

First, obtain the ARQoid JAR and its dependencies.  This is easily accomplished using the download page on the ARQoid Wiki and obtaining the latest ARQoid ZIP file.  Unzip the downloaded archive.   Since I’m discussing an Android application I’d expect that you would have created an Android project and that it contains a libs directory where the JAR files should be placed.

Second, add the JAR files to the classpath for your Android project.  I use the ADT plugin for Eclipse to do Android development.  So to add the JARs to my project I choose the Project menu item, select Properties, choose Build Path, select the Libraries tab, click the Add JARs… button, navigate to the libs directory, select the JAR files and click OK on the open dialogs.

Third, setup a minimal Android project.  The default layout, with a small change to its definition will work fine.

Overview

Now we are ready to write the code that uses ARQoid to access some data.  For this first blog entry I’ll focus on a trivial query against a SPARQL endpoint.  There would be some small differences if we wanted to query a local model or a remote data set.  Those will be covered in follow-on entries.

Here is a list of the ARQoid classes we will be using for this initial example:

• com.hp.hpl.jena.query.Query – represents the query being executed
• com.hp.hpl.jena.query.Syntax – represents the query syntaxes supported by ARQoid
• com.hp.hpl.jena.query.QueryFactory – creates a Query instance based on supplied parameters such as the query string and syntax definition
• com.hp.hpl.jena.query.QueryExecution – provides the service to  execute the query
• com.hp.hpl.jena.query.QueryExecutionFactory – creates a QueryExecution instance based on supplied parameters such as a Query instance and SPARQL endpoint URI
• com.hp.hpl.jena.query.ResultSet – represents the returned data and metadata associated with the executed query
• com.hp.hpl.jena.query.QuerySolution – represents one row of data within the ResultSet.

We’ll use these classes to execute a simple SPARQL query that retrieves some data associated with space exploration.  Talis provides an endpoint that we can use to access some interesting space exploration data.  The endpoint is located at http://api.talis.com/stores/space/services/sparql.
The query we will execute is:

SELECT ?dataType ?data
WHERE {
  <http://nasa.dataincubator.org/launch/1961-012> ?dataType ?data.
}

This query will give us a little information about Vostok 1 launched by the USSR in 1961.

Create the Query instance

We begin by creating the Query instance using the QueryFactory.

// Create a Query instance
Query query = QueryFactory.create(queryString, Syntax.syntaxARQ);

This code assumes that the query given earlier has been assigned to the String  variable queryString

Create the QueryExecution instance

We next create a QueryExecution using the QueryExecutionFactory.

// This query uses an external SPARQL endpoint for processing
// This is the syntax for that type of query
QueryExecution qe = QueryExecutionFactory.sparqlService(sparqlEndpointUri, query);

This code assumes that the Talis endpoint mentioned above has been assigned to the String variable sparqlEndpointUri.

Execute the query and obtain the ResultSet instance

We are now ready to actually execute the query and obtain the ResultSet instance.

// Execute the query and obtain results
ResultSet resultSet = qe.execSelect();

The resultSet variable now provides us access to the query results.

Retrieve the column names

A useful piece of metadata in the ResultSet instance is the list of column names.  These will be based on the information requested in the SELECT clause.  ARQoid can return them as a List<String>.

// Get the column names (the aliases supplied in the SELECT clause)
List<String> columnNames = resultSet.getResultVars();

The columnNames List will contain the aliases given in the SELECT clause.

Iterate through the resulting rows

We can now iterate through the resulting rows, asking for each row’s data which is represented as a QuerySolution instance.

// Iterate through all resulting rows
while (resultSet.hasNext()) {
  // Get the next result row
  QuerySolution solution = resultSet.next();

The solution variable will contain the current result row’s data.

Obtain the data for a row and column

To actually access the data you can request a specific column name from the QuerySolution instance.  However, you need to know whether the data is null, a literal value or a URI.  The following code performs the necessary tests and then prints the data to the standard output (in the downloadable code it will be presented on the Android device’s screen).

// Data value will be null if optional and not present
if (solution.get(var) == null) {
  System.out.println("{null}");
// Test whether the returned value is a literal value
} else if (solution.get(var).isLiteral()) {
  System.out.println (solution.getLiteral(var).toString());
// Otherwise the returned value is a URI
} else {
  System.out.println(solution.getResource(var).getURI());
}

From the code, above, you can see that in order to access the basic data value you use the get(String) method which expects the column name to be passed.  This will return null if there is no data associated with this row and column.  If there is a value, the method isLiteral() may be called to test whether the data is a literal.  If it is not than it will likely be a URI.  The URI can be accessed by calling the getResource(String) method, passing the column name, and calling the getURI() method on that value.

Close the QueryExecution instance

The last step is important.  The QueryExecution instance should be cleaned up by calling its close() method.

// Important - free up resources used running the query
qe.close();

These are the basic steps you need to carry out in order to execute a SPARQL query against a SPARQL endpoint using ARQoid.  Here is a screen shot of the Android emulator executing the example just covered.

Notes

A few other items for completeness.  First, remember to add the INTERNET permission to your Android manifest (<uses-permission android:name=”android.permission.INTERNET”/>).  Failure to do so will lead to an ARQoid failure when it tries to access the remote endpoint.  The stack trace will indicate a failure to access the URI – it won’t mention that there is a permission issue.

Also, depending on your query, you may be trying to access a lot of data which could be time consuming and also could cause issues with small memory devices.  You may limit the number of rows returned and set a starting row for the results.  This allows you to create a sliding window in your application by only pulling a few results and then allowing the user to ask for more.  These values are set on the Query instance.  Once you have the Query from the QueryFactory you can use these methods.

// Limit the number of results returned
// Setting the limit is optional - default is unlimited
query.setLimit(10);

// Set the starting record for results returned
// Setting the limit is optional - default is 1 (and it is 1-based)
query.setOffset(11);

The limit and offset given above would cause ARQoid to return 10 records, starting with the 11th one found. If there were fewer than 11 results then no records would be returned.

Conclusion

Hopefully if you are trying to use ARQoid this will give you a quick template to leverage the basic features of the engine.  In the future I’ll expand on this by adding other data sources as well as using the URIs returned by the query to create a richer result for the user.

Remember that you may download the sample code if you want to see the working Android demonstration application described here.  Also, you can install Sparql Droid which contains a variety of sample SPARQL queries that use the local model, SPARQL endpoints, RDF data sources as well as demonstrating federated queries.

If you have questions or comments about this topic please add them to this post or send them to me via my contact page.

Business rules, decision engines, BPM, complex event processing (CEP), these all invoke images of computers making speedy decisions to the benefit of our businesses.  The infrastructure, technologies and software that provide these solutions (SOA, XML schemas, rule engines, workflow engines, etc.) support the decision automation process.  However, they don’t know what decisions to make.

The BPM-related components we acquire provide the how of decision making (send an email, route a claim, suggest an offer).  Learning, supported by data analytics, provides a powerful path to the what and why of automated decisions (send this email to that person because they are at risk of defecting, route this claim to that underwriter because it looks suspicious, suggest this product to that customer because they appear to be buying these types of items).

I’ll start by outlining the high level journey from data to rules and the cyclic nature of that journey.  Data leads to rules, rules beget responses, responses manifest as more data, new data leads to new rules, and so on.  Therefore, the journey does not end with the definition of a set of processes and rules.  This link between updated data and the determination of new processes and rules is the essence of any learning process, providing a key function for the cognitive corporation.

The following image depicts the overall process of using data analytics to take corporate information and derive new business processes and rules, hence learning.  I will refer to the numbered items throughout this post.

Data Analytics is a general term and there are many subtleties within such a broad field.  In this case I’m focused on the machine (computer) learning aspect of analytics.  I’m skipping past the need to identify key data sources, create canonical definitions, and normalize information.  Overused, yet accurate, the phrase, “Garbage In, Garbage Out” applies to data analytics as much as any other aspect of computing.  The relevant data must be effectively organized before proceeding (in the diagram this is depicted as items 1, 2 and 3).

What is machine learning?  It is a way that computers can be used to look at data and find patterns that we don’t realize exist.  There are a variety of algorithms that allow computers to be used for this purpose.  Some are very difficult to understand and some are quite simple.  They each have strengths and weaknesses.  Not all work well for a given type of data or analytics.

Therefore, the first step for leveraging data analytics (on clean data, of course) is to understand the computer’s perception of that data at a high-level.  This is known as data preprocessing and contains within it tasks that are necessary in order to find actionable information within the data (diagram item 4).  The tasks can be outlined as: Aggregation, Sampling, Dimensionality Reduction, Feature Subset Selection, Feature Creation, Discretization, and Feature Transformation.  I will explore each of these tasks in future postings.

Once the data is understood and we know how to structure it for automated analytics (diagram items 5 and 6), we need to run it through machine learning algorithms (diagram item 7).  These are the secret sauce of many data analytics tools.  The algorithms involve a variety of mathematical approaches to identifying relationships within the data.  What isn’t always easy to understand is why the analytics are identifying certain relationships.  Some algorithms make this harder than others to discern.  As with data preprocessing’s steps, I’ll delve into these algorithms in future posts.

Fundamentally, the output from this type of tool is a predictive model (diagram item 8) that can take new data and predict the outcomes.  Assuming we have focused on something pertinent to our operations, retaining customers perhaps, then we want the computer to find a way to predict which customers are likely to defect.  If such a model is tested and found to provide accurate results then we are confident that within its black box it “knows” something about our defecting clients that we might be able to use in order to improve retention.

It is the next step, being able to use the model to derive a business rule change, which requires human intervention.  The task is to root out the underlying cause(s), which the computer has found, in a way that we are able to understand (diagram item 9) and, more importantly, that we may may act upon (diagram item 10).  This is the crux of leveraging data analytics to improve business operations.

To successfully leverage the data analytics findings (items 9 and 10) a business needs three things.  First, it must have a formal understanding of the processes used within the analytic environment.  Experts must be able to translate the predictive model back to the actual source data that is pertinent to the predictions.

Second, there must be an open and collaborative environment that accepts unanticipated data relationships, working to define them in business terms – not looking to dispute or discredit them. Often, people who “know the business” cannot accept what the data analytics results are showing.  The findings, often by definition, are at odds with commonly accepted “truths.”  This is part of the power of using these tools, they aren’t beholden to our mental baggage and, by design, they think outside of our business-knowledge-based box.

Third, the business must have an agile environment that can make changes to the underlying processes and rules culled from the model.  This permits the patterns and causes found by the data analytics tool to manifest as timely action within the business’ operation.  Therefore, there must be business leaders empowered to change business processes and rules, as well as an IT infrastructure that supports implementing rapid rule and process changes (diagram step 11).

The faster an organization can take data analytics results and update business operations (rules and processes), the better its ability to respond to changing business environments.  Such a capability provides a key differentiator for any business.  More and more companies are finding themselves competing in ever more commoditized markets.  It is an organization’s unique ability to use automation as a way to quickly personalize and flex to specific situations that allows it to be more productive, more responsive and ultimately more successful.

This process is not simple nor is it completely automatic.  It consistently requires manual effort and tuning.  Machines don’t do the thinking, they simply provide leaders with actionable information by which conclusions may be drawn and decisions may be made.

Some business use cases for data analytics tools are easier to implement than others.  For instance, predictive models around actuarial data lead to scoring rules in a business rule environment fairly directly.  The relationships between buying patterns and social media and their impact on business processes and rules are not as straightforward to map.  The rewards, however, are meaningful and worthwhile as long as the correct data is being used to drive the correct decisions.

If you are exploring data analytics as an input to business rule and process improvements I’d be interested in hearing about your experiences and thoughts.  Also, as I mentioned at the beginning, this post is meant to introduce the high-level concepts.  Future postings will dive deeper into the individual processes, tools and techniques that were touched upon.

I understand the desire to rely on unit testing and its ability to find issues and prevent regressions.  For TDD, I’ll need to write separately.  Fundamentally I’m a believer in white box testing.   Black box approaches, like TDD, seem to be of relatively little value to the overall quality and reliability of the code.  Meaning, I’d want to invest more effort in white box testing than in black box testing.

I’m somewhat jaded, being concerned with the code’s security, which to me is strongly correlated with its reliability.  That said, I believe that unit testing is much more constrained as compared to formal reviews.  I’m not suggesting that unit tests be skipped, rather that we understand that unit tests can catch certain types of flaws and that those types are narrow as compared to what formal reviews can identify.

Here is how I view the constraints around unit testing:

First, unit tests are normally implemented by the same developer who created (or will create) the code.  This is a huge constraint on the brain power (BP) being applied to the quality of the code.  In this case BP=1.  If the unit tests are created by a separate individual then BP>1 and, I argue, BP<2 because the tester is focused on the micro level (the unit level of code).

Second, unit testing is often used to only check that the code does what it is supposed to do.  This means unit tests don’t often check whether code does things it isn’t supposed to.

Third, unit tests don’t consider maintainability of the code, or really any macro-level concerns.  Passing unit test doesn’t mean that the code is easy to understand or logically organized.  I agree that single-purpose methods and well written code are easier to test, but bad code can be tested and even meet high coverage requirements.

The reason I believe that the study’s results (discussed in the previous post) are probably still accurate is that many software fundamentals are no different now than in 1986.  We used unit testing frameworks (sometimes called scaffolds) and leveraged unit tests to prevent regressions.  In some ways we had to be more concerned with the completeness of our test scenarios in 1986 due to the overhead of modifying an application.  Developers today can be somewhat cavalier with code changes since our system environments tend make distribution of updates easier.

I agree that new languages, paradigms and environments have not reduced the need for unit testing, quite the contrary.  We have more options and threats to deal with.  However, we can more quickly make and distribute an updated application than we could in 1986.

Considering the breadth of what can be identified, formal design and code reviews go beyond simply checking that for some inputs a valid output is calculated.  Instead, formal reviews apply BP=T(RE%) where T is the team size and RE% is the review effort percentage the team invests.

These reviews catch maintenance issues, identify best practices violations (beyond what a style checker would do), find opportunities for refactoring (reducing code, reducing complexity, etc.) and serve as a way for the team to increase its overall expertise through interactions and discussion.  Clearly, this represents a much broader value set than that provided by unit testing.

As I said in the original posting, this thinking isn’t a defense to skip unit testing in favor of formal reviews.  (I clamor for unit tests and high coverage metrics.)  Instead, my message is a call to understand that you need both (and other types of testing and inspections), realizing that the formal reviews catch a more comprehensive set of issues than unit testing.  Time and again I have seen organizations leveraging unit testing but not formal reviews.  This means that their RE% is zero.  For me this is a poor investment choice.

Perhaps it would be worth doing a small scale study to see if our shop has a similar experience?  It might be quite informative for the whole team.

At one point we were reviewing some code that had several error handling blocks that were not being covered in the unit tests.  These blocks were, arguably, unlikely or impossible to reach (such as a Java StringReader throwing an IOException).  There was some discussion by the team about the necessity of mocking enough functionality to cover these blocks.

Although we agreed that some of the more esoteric error conditions weren’t worth the programmer’s time to mock-up, it occurred to me later that we were missing an important point.  What mattered was that we were holding a formal code review and looking at those blocks of code.

Let me take a step back.  In 1986, Capers Jones published a book entitled Programming Productivity.  Although dated, the book contains many excellent points that cause you think about how to create software in an efficient way.  Here efficiency is not about lines of code per unit of time, but more importantly, lines of correct code per unit of time.  This means taking into account rework due to errors and omissions.

One if the studies presented in the book relates to identifying defects in code.  It is a study whose results seem obvious when we think about them.  However, we don’t always align our software development practices to leverage the study’s lessons and maximize our development efficiency.  Perhaps we believe that the statistics have changed due to language construct, experience, tooling and so forth.  We’d need similar studies to the ones presented by Capers Jones in order to prove that, though.

Below are a few of the actions from the book’s study of defect detection approaches.  I’ve skipped the low end and high-end numbers that Caper’s includes, simply giving the modes (averages) which are a good basis for comparison:

Based on this data, we see that formal reviews of the design and code are much more effective at finding defects than unit testing.  In this case, unit testing is focused on branch coverage.  Obviously, changing the testing expectations, such as applying domain testing concepts, might change the effectiveness. My guess is that such a change would not bring unit testing on par with formal code reviews but would improve its performance, at a cost of productivity – since domain testing takes more rigor to apply.

As Capers points out, these percentages can’t be added together.  In other words, there is an intersection of the defects found by these methods.  In his studies these methods found some of the same and some unique defects.  Therefore, this data does not provide a defense to skip unit testing.  Instead, each defect detection method should be used in a complementary fashion.  The statistical data does inform us regarding level-of-effort decisions.  Also, we can use this information to help define where to cut corners, if necessary.

If we have to reduce the time we spend in the testing phase, we would do better to give up some unit testing in favor of keeping more formal code reviews.  This isn’t always the way we approach such situations.  It is easy to skip the formal reviews, believing that the time would be better spent by developers creating more tests for their code.  Apparently this is not the case.

I think it is valuable to periodically remind ourselves about studies like these as a way to make sure we aren’t falling into bad habits or being misled by incorrect assumptions.  Also, as teams bring on new members, it is good to review our practices, understand the relevant literature and actively discuss why we do what we do (and whether it needs to change).

A parting thought about statistics like the ones quoted here.  They are an average from a very broad array of problem types and development shops.  As Boris Beizer has pointed out, each shop has a “bug fingerprint”.  The bug fingerprint is impacted by things like the technologies being used, types of applications being developed and the experience of the team members.

Further, different types of bugs are better identified through different detection methods (branch tests, domain tests, code inspections, etc.).  Therefore, the optimal set of testing approaches for a given group of developers will differ somewhat from the average.  I’ll discuss this in a future post.

What does your team do in regards to testing and inspections?  Are there other techniques you find useful for identifying software defects?  I welcome your comments and feedback.

As I’ve placed these separate projects into a common context of my own, I’ve concluded that the long term goal of leveraging information technology must be for it to support cognitive processes.  I don’t mean that the computers will think for us, rather that IT solutions must work together to allow a business to learn, corporately.

The individual tools that we utilize each play a part.  However, we tend to utilize them in a manner that focuses on isolated and directed operation rather than incorporating them into an overall learning loop.  In other words, we install tools that we direct without asking them to help us find better directions to give.

Let me start with a definition: similar to thinking beings, a cognitive corporation™ leverages a feedback loop of information and experiences to inform future processes and rules.  Fundamentally, learning is a process and it involves taking known facts and experiences and combining them to create new hypothesis which are tested in order to derive new facts, processes and rules.  Unfortunately, we don’t often leverage our enterprise applications in this way.

We have many tools available to us in the enterprise IT realm.  These include database management systems, business process management environments, rule engines, reporting tools, content management applications, data analytics tools, complex event processing environments, enterprise service buses, and ETL tools.  Individually, these components are used to solve specific, predefined issues with the operation of a business.  However, this is not an optimal way to leverage them.

If we consider that these tools mimic aspects of an intelligent being, then we need to leverage them in a fashion that manifests the cognitive capability in preference to simply deploying a point-solution.  This involves thinking about the tools somewhat differently.

As I consider the vision of empowering a learning corporation, I break enterprise tools into 6 general areas.  These areas provide a portion of what is required in order to survive and grow in an environment.  These 6 areas are: Think, Communicate, Act, Know, Sense and Learn.  They are in no particular order since the absence of any will prevent meaningful growth from occurring.

I’ll be expanding on this concept over time, drilling into each area and the solutions we often use to meet that need.  Depicted below is a very high-level diagram of these areas with the categories of products or functions I place in each.  The categories are broad and I’ve chosen key terms to represent a lot of different tools and techniques.

This framework provides an infrastructure, similar to how a person is made up of a set of systems.  However, it is the information that is shared between systems that becomes the key to actually supporting the idea of a thinking and learning system.  There are assumptions we make as we design enterprise systems that will limit our ability to empower a cognitive corporation™.

For instance, we may limit our definition of valuable data to the information collected by our on-line systems.  A broader interpretation begins to consider the process steps followed and the rules executed for a given interaction as data.  There are many places where such assumptions and limitations will interfere with gaining a true advantage from our enterprise solutions.

We must change our view of the way that a business benefits through the use of IT.  The focus on feature sets for a specific type of problem must be replaced with a focus on the whole set of systems and how they interrelate and may be leveraged to benefit business growth – in multiple dimensions.  This will allow us to make an order-of-magnitude jump in the value derived from IT investments.

Therefore, an enterprise architecture goes beyond the role that systems will play, it must define how the systems will interact within the entire enterprise.  We are then in a position to leverage these powerful tools to create a cognitive corporation™.