Archive

Archive for 2009

ArcGIS Online Moving to Google/Bing Tiling Scheme

November 23rd, 2009 Comments off

In an interesting move, ESRI is changing the tiling scheme of ArcGIS Online services to match the tiling scheme used by Google Maps and Bing Maps.

By the end of the year ArcGIS Online services are planned to be published in the Mercator projection with 256 x 256 pixel tiles, simplifying the process of creating mashups with Google Maps and Bing Maps.

Read more about this change on ESRI’s ArcGIS Server Blog.

Categories: GIS, Software Development Tags: , , , ,

Bootstrapping the ArcGIS Server JavaScript API

October 26th, 2009 1 comment

On a project my team is currently working on with we came across the need to bootstrap the ArcGIS Server JavaScript API.  What I mean by ‘bootstrapping’ is the ability to load the JavaScript API during script execution after page load without loading it declaratively in the <head> of the HTML page.

To better describe the need for this here is a summary of our requirements:

  1. We wanted the ability to embed our live maps (and map related capabilities) as widgets in other HTML pages throughout our company website
  2. We wanted the ability to embed these live maps as a stand-alone snippets of HTML without having to modify the page <head>.  This would allow content authors to embed maps without having access to modify anything but a container element on the page.
  3. We wanted the ability to wrap these up in a container (in our case this will probably be a DotNetNuke module)
  4. We didn’t want the content author to have to know anything about the actual implementation of the map engine.  This future-proofs the implementation as the content author does not have to refer to a specific version of the JavaScript API or even know that the JavaScript API is the engine behind the map.
  5. We didn’t want the loading of the embedded map to have direct impact on the loading of the parent page (asynchronous loading)
  6. We didn’t want to deal with iframes for embedded map widgets

What we wanted was the ability to include a snippet like the following in any page and get a fully dynamic map widget (Note: I have renamed the objects/namespaces to make this description generic. myMapApi is used as the base namespace for these examples):

<!-- Start map widget -->
<script language="javascript" src="http://www.website.com/maps/api.js">
<div id="map" style="width: 600px; height: 400px;"></div>
<script type="text/javascript">
    var map = new myMapApi.Map("map");
</script><br />
<!-- End map widget -->

The ArcGIS Server Javascript API is based on Dojo. When the Javascript API is normally loaded (as a script in the page header) the script will automatically load Dojo and Dojo will then bootstrap the resources that it needs. Dojo includes the ability to load scripts on the fly (using dojo.require) which is the method that is used to load the modules that are required for the current page.

Because Dojo is bootstrapped and modules can be loaded on demand, Dojo provides an addOnLoad method which allows you to ensure that everything has been loaded before proceeding with the execution of code that depends on Dojo components.

The addOnLoad method accepts a callback argument that is used to ‘call back’ when the following conditions are met:

  1. The DOM is loaded
  2. All Dojo modules are loaded (and their recursive dependencies)

If you take a look at the basic map sample from the Javascript API docs you will see the general sequence of events:

  1. Add the Javascript API script reference to the page head
  2. Run dojo.require for all necessary Dojo modules
  3. Create an init method to be called back when everything is loaded
  4. Pass this init method to dojo.addOnload to kick off the execution of the script when the page is loaded.

This works great when you include the ArcGIS Server Javascript API directly in the page head but we wanted to do this by wrapping the Javascript API in our own application API. As noted above, consumers of our maps would then not have to know anything about the ArcGIS Server Javascript API or Dojo.

To get around the async loading that occurs when scripts are included dynamically we implemented a mechanism that is similar to the Dojo addOnLoad.

Here is a sample snippet that would appear in a web page to include our map widget:

<!-- Start map widget -->
<script language="javascript" type="text/javascript" src="api.js"></script>	
<div id="map" style="width: 600px; height: 400px;"></div>
    <script type="text/javascript">
        function init() {
            var map = new myMapApi.Map("map");
        }
        myMapApi.onLoad(init);
</script>
<!-- End map widget -->

Similar to Dojo, the onLoad method accepts a callback method that our API can call when everything it needs has been loaded. The code in our API can then call the callback whenever the ArcGIS Javascript API has been loaded dynamically and whenever Dojo is fully loaded.

The basic challenge at this point was to determine how to load the ArcGIS Server Javascript API dynamically and then determine when it was fully loaded so that the callback method could be called.

Here is what we came up with:

// Create the namespace
if (myMapApi == null || typeof (myMapApi) == 'undefined') { var myMapApi = {}; }

// ESRI Javascript API bootstrapper
(function() {

    // Configure Dojo onLoad so that it works after the page's
	// onload event has already occurred
    djConfig = {
        afterOnLoad: true,
        addOnLoad: function() {
            myMapApi.scriptsLoaded = true;
            if (typeof (myMapApi.onLoadCallback) != 'undefined') {
                myMapApi.onLoadCallback();
            }
        }
    };

    // Dynamically load the Javascript API
    var scriptElement = document.createElement('script');
    scriptElement.type = 'text/javascript';
    scriptElement.src = 'http://serverapi.arcgisonline.com/jsapi/arcgis/?v=1.5';
    document.getElementsByTagName('head')[0].appendChild(scriptElement);
	
	// Dynamically load a Dojo stylesheet
    var linkElement = document.createElement("link");
    linkElement.type = 'text/css';
	linkElement.rel = 'stylesheet';
	linkElement.media = 'screen';
    linkElement.href = 'http://serverapi.arcgisonline.com/jsapi/arcgis/1.5/js/dojo/dijit/themes/tundra/tundra.css';
    document.getElementsByTagName('head')[0].appendChild(linkElement);

})();

// Create an onLoad event to call the client back on when
// dependant scripts have been loaded 
myMapApi.onLoad = function(onLoadCallback) {
    myMapApi.onLoadCallback = onLoadCallback;
    if (myMapApi.scriptsLoaded) { myMapApi.onLoadCallback(); }
}

// The Map class creates and loads the actual map in the parent container
myMapApi.Map = function(containerId) {
	
	// Get the container in which to create the map
	var container = dojo.byId(containerId);
	
    // Create a div for the map
    var mapDiv = document.createElement("div");
    mapDiv.className = "tundra";
    mapDiv.id = "mapdiv";
	container.appendChild(mapDiv);
    mapDiv.style.width = container.offsetWidth;
    mapDiv.style.height = container.offsetHeight;

    // Create the map
    var map = new esri.Map("mapdiv");

	// Add a service to the map
	var service = "http://server.arcgisonline.com/ArcGIS/rest/services/ESRI_StreetMap_World_2D/MapServer";
    var tiledMapServiceLayer = new esri.layers.ArcGISTiledMapServiceLayer(service);
    map.addLayer(tiledMapServiceLayer);
	
	return this;
}

Here is a quick breakdown of how this works:

Line 2: Set up the myMapApi namespace.

Line 5: This code is implemented entirely as an immediately executable anonymous function to avoid creating any variables within the scope of the page. You can see a description of this technique here.

Line 9-17: The djConfig object is the key to getting this technique to work. The default behavior of addOnLoad is to fire when all dependencies are loaded (including the page DOM). As a result, if the DOM is already loaded the addOnLoad event will not occur. In many cases with a widget (sometimes depending on the browser) the DOM may already be considered to be loaded and the addOnLoad will not fire. The afterAddOnLoad property allows this behavior to be ignored and addOnLoad will fire whenever Dojo dependencies are loaded.

Line 11-16: This anonymous functions defines the addOnLoad behavior. In our case we note that the Javascript API is loaded and then we call the callback (if it is has been provided). Depending on the timing of the page load the callback may or may not have been provided at this point.

Line 19-23: Dynamically load the JavaScript API

Line 26-31: Dynamically load the tundra theme

Line 37-40: This anonymous function allows the consuming page (the page the contains the widget) to provide the callback function. A reference to the callback is kept in the event that the JavaScript API has not been loaded yet (at which point it will then be called on line 14. If the Javascript API is already loaded (as evaluated on line 39, the callback will be called.

Line 43-65: This class creates the div that will be used for the map, news up an esri.Map, and loads a map service into the div.

And that is basically it..

Here is the output from Firebug showing the loading sequence:

Bootstrapping the ESRI Javascript API - Firebug Output

I have also posted a working example of this technique. This has been tested to work in IE 7, FireFox 3.x and Chrome 3.x. If you find that it works (or doesn’t work) please let me know. Suggestions or other ideas are also welcome!

Categories: GIS, Software Development Tags: , , , , , ,

Certified ScrumMaster Training

October 19th, 2009 Comments off

Certified ScrumMaster

I recently had the opportunity to attend a Certified ScrumMaster (CSM) training/certification course at the Rally Software Development Headquarters in Boulder, Colorado.  I have been using Scrum for over a year now and this was an excellent opportunity to refine skills and to get real-world questions answered.

If you follow the activities of the Scrum Alliance (the group that administers the Scrum certification process) you will know that a test is now required to obtain ScrumMaster Certification.

Previously you only had to take the course to become certified.  As of October 1, 2009 the test is in ‘beta testing’ to allow the Scrum Alliance to validate the quality of the test questions before implementing the final pass/fail test.  The test is based on the Scrum Guide.

The course that I attended was instructed by Tamara Sulaiman.  The training was excellent and Tamara did an excellent job of illustrating the concepts with real-world examples. Tamara’s in-depth knowledge and hands-on experience added a lot of depth to the course.

Key Take-Aways

Product Owner Role

The Product Owner Role is critical to the success of a Scrum project.  The Product Owner must be very involved in the project and must be highly available to the team.

Sprint Retrospective

Sprint Retrospectives are the key to frequent inspection and adaptation. This ultimately is the whole point of Scrum – the ability to manage and control projects by inspecting and adapting frequently. Insufficient use the Retrospective was identified as one of the primary reasons for Scrum failure.

Product Backlog Grooming

I hadn’t actually heard of this specific process and I have frequently encountered challenges ensuring that the Product Backlog is “ready” prior to the next Sprint. While not a formal part of Scrum, Product Backlog Grooming is the activity of cleaning the Product Backlog during the sprint.

This ensures that the Delivery Team and the Product Owner know what is coming next (in advance) and ensures that the Product Backlog is in a clean state prior to each Sprint Planning meeting.

It was suggested that about 5% of the sprint capacity be allocated to this activity.  Product Backlog Grooming meetings should be scheduled in the same way that the other Scrum meetings are scheduled.

Levels of Agile Planning

This was another area that is not formally a part of Scrum but that I found to be very helpful.  The idea is that agile planning happens at multiple levels and that ‘sprinting’ on a Scrum project is only one of those levels.  Here is a breakdown of the different levels of planning that were suggested:

  • Product Vision – Defining the high-level vision for a product (By the Product Owner approx. 1-2 times per year)
  • Roadmap – Defining the product roadmap and creating the initial Product Backlog (By the Product Owner/Architect approx. 2-4 times per year)
  • Release Planning – Planning of product releases (By the Product Owner, Team & Stakeholders approx. 3-4 times per year)
  • Sprint Planning – The Scrum process of planning for each sprint (By the Product Owner and Delivery Team for every sprint)
  • Daily Scrum – The Scrum process of the daily stand-up meeting (By the Delivery Team)

Here are some additional resources about the topic of agile planning:

Five Levels of Planning: To Agility and Beyond! (RallyDev, 2006)

Scaling Agile Processes: Five Levels of Planning (Agile Journal, 2007)

Resources

Several books were mentioned during the training.  Here is a listing for reference:

The Software Product Manager’s Bridge to Agility (Michele Sliger, Stacia Broderick, 2008)

Scrum and XP from the Trenches (Henrik Kniberg, 2007)

Agile Retrospectives (Esther Derby, Diana Larsen, Ken Schwaber, 2006)

Agile Estimating and Planning (Mike Cohn, 2005)

User Stories Applied (Mike Cohn, 2004)

Teamwork is an Individual Skill: Getting Your Work Done When Sharing Responsibility (Christopher M. Avery, Meri Aaron Walker, Erin O’Toole Murphy, 2001)

Categories: Agile, Software Development Tags: , , , ,

ArcGIS Server JavaScript API 1.5 Released

October 2nd, 2009 Comments off

An update to the ArcGIS Server Javascript API has been released.

Of the most notable updates, this release fixes the Firefox 3.5 picture marker bug and updates the Dojo toolkit to version 1.3.2.

Read the what’s new for more details.

Categories: GIS Tags: , , , , ,

Quickly Generate C# Data Objects from XML

September 19th, 2009 1 comment

Ever had a need to read an existing XML document in a .NET app but didn’t want to deal with XPath queries or navigating the DOM?  Wouldn’t it be easier if you could just use data objects instead?  Enter Xsd.exe..

Xsd.exe is a free tool that ships with Visual Studio (including the Express editions) that allows you to “generate XML schema or common language runtime classes from XDR, XML, and XSD files, or from classes in a runtime assembly”.

Or to put it a different way, one of the abilities of Xsd.exe is the ability to auto-generate classes from an XML or XSD document.  With these classes in hand you can then deserialize an XML document at runtime and access the document without having to deal with the underlying XML.

Here’s how:

Step 1: Create the Schema Definition

First you’ll need to have a copy of the XML file that you intend to read.  If you already have an XSD schema file for the XML document skip to Step 2.

Here is an sample XML file provided with the MSXML SDK.   I have removed some of the book entries for brevity and saved this to a file named ‘books.xml’.

<?xml version="1.0"?>
<catalog>
   <book id="bk102">
      <author>Ralls, Kim</author>
      <title>Midnight Rain</title>
      <genre>Fantasy</genre>
      <price>5.95</price>
      <publish_date>2000-12-16</publish_date>
      <description>A former architect battles corporate zombies,
      an evil sorceress, and her own childhood to become queen
      of the world.</description>
   </book>
   <book id="bk106">
      <author>Randall, Cynthia</author>
      <title>Lover Birds</title>
      <genre>Romance</genre>
      <price>4.95</price>
      <publish_date>2000-09-02</publish_date>
      <description>When Carla meets Paul at an ornithology
      conference, tempers fly as feathers get ruffled.</description>
   </book>
   <book id="bk108">
      <author>Knorr, Stefan</author>
      <title>Creepy Crawlies</title>
      <genre>Horror</genre>
      <price>4.95</price>
      <publish_date>2000-12-06</publish_date>
      <description>An anthology of horror stories about roaches,
      centipedes, scorpions  and other insects.</description>
   </book>
   <book id="bk110">
      <author>O'Brien, Tim</author>
      <title>Microsoft .NET: The Programming Bible</title>
      <genre>Computer</genre>
      <price>36.95</price>
      <publish_date>2000-12-09</publish_date>
      <description>Microsoft's .NET initiative is explored in
      detail in this deep programmer's reference.</description>
   </book>
</catalog>

Xsd.exe can generate classes directly from the XML document but I prefer to create an schema definition first to confirm that the data elements have been interpreted correctly.  If an XML file has no value for a particular element, Xsd.exe has no way of inferring the data type.  In these cases you may need to tweak the schema document before generating the data classes.

Ok, so let’s go ahead and create the XSD.  Xsd.exe is typically located at C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin.

Here’s the command to create the XSD schema from the XML file:

>xsd.exe books.xml

Microsoft (R) Xml Schemas/DataTypes support utility
[Microsoft (R) .NET Framework, Version 2.0.50727.42]
Copyright (C) Microsoft Corporation. All rights reserved.
Writing file 'books.xsd'.

Here is the content of books.xsd:

<?xml version="1.0" encoding="utf-8"?>
<xs:schema id="catalog" xmlns="" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
  <xs:element name="catalog" msdata:IsDataSet="true" msdata:UseCurrentLocale="true">
    <xs:complexType>
      <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element name="book">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="author" type="xs:string" minOccurs="0" msdata:Ordinal="0" />
              <xs:element name="title" type="xs:string" minOccurs="0" msdata:Ordinal="1" />
              <xs:element name="genre" type="xs:string" minOccurs="0" msdata:Ordinal="2" />
              <xs:element name="price" type="xs:string" minOccurs="0" msdata:Ordinal="3" />
              <xs:element name="publish_date" type="xs:string" minOccurs="0" msdata:Ordinal="4" />
              <xs:element name="description" type="xs:string" minOccurs="0" msdata:Ordinal="5" />
            </xs:sequence>
            <xs:attribute name="id" type="xs:string" />
          </xs:complexType>
        </xs:element>
      </xs:choice>
    </xs:complexType>
  </xs:element>
</xs:schema>

You will notice that Xsd.exe interpreted the publish_date element as a string. This would be more useful as a date data type. This can be done by changing the type from xs:string to xs:date like so:

<xs:element name="publish_date" type="xs:date" minOccurs="0" msdata:Ordinal="4" />

Step 2: Generating the Data Classes

Now that we have a valid XSD file and we have reviewed it for accuracy, let’s generate the C# classes.

Here’s the command:

>xsd.exe -c books.xsd

Microsoft (R) Xml Schemas/DataTypes support utility
[Microsoft (R) .NET Framework, Version 2.0.50727.42]
Copyright (C) Microsoft Corporation. All rights reserved.
Writing file 'books.cs'.

Note that there are additional parameters that you can use to set the namespace etc. but this will do for demonstration purposes.

Here is the content of books.cs:

//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool.
//     Runtime Version:2.0.50727.3082
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

using System.Xml.Serialization;

// 
// This source code was auto-generated by xsd, Version=2.0.50727.42.
// 


/// <remarks/>
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd", "2.0.50727.42")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(AnonymousType=true)]
[System.Xml.Serialization.XmlRootAttribute(Namespace="", IsNullable=false)]
public partial class catalog {
    
    private catalogBook[] itemsField;
    
    /// <remarks/>
    [System.Xml.Serialization.XmlElementAttribute("book", Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
    public catalogBook[] Items {
        get {
            return this.itemsField;
        }
        set {
            this.itemsField = value;
        }
    }
}

/// <remarks/>
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd", "2.0.50727.42")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(AnonymousType=true)]
public partial class catalogBook {
    
    private string authorField;
    
    private string titleField;
    
    private string genreField;
    
    private string priceField;
    
    private System.DateTime publish_dateField;
    
    private bool publish_dateFieldSpecified;
    
    private string descriptionField;
    
    private string idField;
    
    /// <remarks/>
    [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
    public string author {
        get {
            return this.authorField;
        }
        set {
            this.authorField = value;
        }
    }
    
    /// <remarks/>
    [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
    public string title {
        get {
            return this.titleField;
        }
        set {
            this.titleField = value;
        }
    }
    
    /// <remarks/>
    [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
    public string genre {
        get {
            return this.genreField;
        }
        set {
            this.genreField = value;
        }
    }
    
    /// <remarks/>
    [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
    public string price {
        get {
            return this.priceField;
        }
        set {
            this.priceField = value;
        }
    }
    
    /// <remarks/>
    [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified, DataType="date")]
    public System.DateTime publish_date {
        get {
            return this.publish_dateField;
        }
        set {
            this.publish_dateField = value;
        }
    }
    
    /// <remarks/>
    [System.Xml.Serialization.XmlIgnoreAttribute()]
    public bool publish_dateSpecified {
        get {
            return this.publish_dateFieldSpecified;
        }
        set {
            this.publish_dateFieldSpecified = value;
        }
    }
    
    /// <remarks/>
    [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
    public string description {
        get {
            return this.descriptionField;
        }
        set {
            this.descriptionField = value;
        }
    }
    
    /// <remarks/>
    [System.Xml.Serialization.XmlAttributeAttribute()]
    public string id {
        get {
            return this.idField;
        }
        set {
            this.idField = value;
        }
    }
}

Step 3: Add the Class to the Solution

So far so good.. Now we need to add the C# class to our Visual Studio solution (copy the class to the solution, right-click the project file, Add > Existing Item..).

At this point we should now have a books.cs file added to our solution and the project should build.

Step 4: Reading the XML File
Almost there! Now we need a little bit of code to deserialize the XML file and new up our data objects.

Here is what that code looks like:

catalog catalog = (catalog)serializer.Deserialize(reader);

XmlSerializer serializer = new XmlSerializer(typeof(catalog));
using (TextReader reader = new StreamReader(@"path\to\xml\file.xml"))
{
     catalog catalog = (catalog)serializer.Deserialize(reader);
}

Now that we have a “newed up” catalog object we’re good to go.. We can now do something like the following:

foreach (catalogBook book in catalog.Items)
{
    System.Console.WriteLine(book.author);
    if (book.publish_date > DateTime.Now.AddMonths(-1))
    {
        // This book was published within the last month
    }
}

Step 5: Cleaning Things Up
At this point things are working but if you are like me you probably aren’t too happy about the auto-generated class and property names.

If you want to clean things up a little here’s an easy way to do it..

The first thing you need to do is add attributes to each of the properties in the auto-generated code. Here is the auto-generated property for the “author” element:

[System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
public string author
{
    get
    {
        return this.authorField;
    }
    set
    {
        this.authorField = value;
    }
}

And here is the updated property (Note that ElementName=”author” has been added). This maps the property (currently named author) to the “author” xml element.

[System.Xml.Serialization.XmlElementAttribute(ElementName = "author", Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
public string author
{
    get
    {
        return this.authorField;
    }
    set
    {
        this.authorField = value;
    }
}

Now we can change the name of the actual property to whatever we want (Right-click, refactor, rename..).

Note that we can do the same with the class names (catalog and catalogBook).

So after a little bit of editing and refactoring here’s how things look.

Catalog catalog = (Catalog)serializer.Deserialize(reader);

foreach (Book book in catalog.Items)
{
    System.Console.WriteLine(book.Author);
    if (book.PublishDate > DateTime.Now.AddMonths(-1))
    {
        // This book was published within the last month
    }
}

Ah, much better!

Other Solutions

As always, there are other ways to do this type of thing (LINQ to XML, LINQ to XSD, xmlobjects etc.) but I have found this simple approach to come in quite handy when I need to get up and running quickly with auto-gen’d data objects from an existing XML file.

Categories: Software Development Tags: , , ,

Building a Smart Irrigation Controller (Part 5)

August 24th, 2009 4 comments

Quick Links: Part 1, Part 2, Part 3, Part 4, Part 5

Conclusions and Next Steps

My ‘smart’ irrigation system has been up and running for about about 6 weeks at the time of this posting.  The ability of the system to increase watering frequency during hot dry days and to scale back during cooler, more humid days is working very well.  I love the idea of not having to mess with the zone timings as time of year changes.

I will be tracking the water consumption over time to compare the usage against my previous irrigation controller.  Based on some preliminary calculations, I expect about a 20-30% reduction in overall water consumption.

The system is working well but I plan on making a few tweaks and improvements based on observations so far.

Next Steps

ET Calculation Adjustment Factor

The ET calculation that I am using assumes clear-sky radiation with a flat surface (no solar or wind obstructions).  As a result, I expect that the ET amounts are slightly overestimated.  I am going to add an adjustment factor to the ET calculations so that I can further optimize the water consumption.

This won’t be too scientific as I simply plan on adjusting this factor according to physical observations of the soil moisture over time.

Irrigation During Cooler Months

I am curious as to how the system will handle the times of year where very little water is required (spring and fall).  I may end up needing to create a second irrigation cycle that applies a smaller amount of water during those times of year so that the irrigation cycles are run more frequently.

This would be easy to do by adding a spring and fall program that would only be active for those time periods.  The allowable water balance deficit would remain the same but the irrigation applied would be less so that the lawn is irrigated more frequently.

I would be able to make those changes within the ISY without any mods required in the online application.

Runoff and Percolation Calculations

Rainfall amounts reported by WeatherBug are recorded daily in the water balance database.  The WeatherBug station measures total daily rainfall but this amount does not consider run-off and percolation below the root zone.  I currently have a cap of 1 inch on the daily rainfall to accommodate for this but this is not a very accurate approach as this doesn’t consider the rainfall rate over the 24 hour period.

For example, consider a moderately heavy rain event that contributes 2 inches evenly over a 24 hour period.  If the soil can absorb .25 inches per hour (a clay soil) then minimal run-off will occur.  In this case the soil moisture capacity will be the limiting factor as anything over the soil moisture capacity will percolate away below the root-zone resulting in a net water contribution that is less than 2 inches.

On the other hand, consider a downpour that contributes 2 inches over a 3 hour period (this happened about a month ago).  In this case the absorption rate of the soil (assume .25 in/hr) will result in roughly 1.25 inches of run-off.  In this case the limiting factor will be the absorption rate of the soil.

The problem that I have currently is that I track the rainfall over a 24 hour period so I am unable to factor the infiltration rate into my calculations.  I plan on tracking rainfall on an hourly basis to optimize this.  If I add the rainfall hourly I can then add calculations that factor run-off and percolation into the water balance equations.  To do this I would add another request type to the online interface and run an hourly program in the ISY to update the water balance.

Hardware Thoughts

I took a leap into Insteon with this project and here are some thoughts on each of the major hardware components that were used.

ISY-99i

The ISY-99i has been very reliable and works as an automation controller should – it just works and does what it is supposed to do.  The ISY is the heartbeat of my Insteon system and I look forward to adding more capabilities to the system in the future.

Universal Devices have provided outstanding customer service and have demonstrated a strong commitment to the product.  The ISY community is very active (forums) and its members are engaged in lots of great projects.

As mentioned previously in this series, Universal Devices expressed an interest in integrating evapotranspiration calculations into the ISY to allow it to function independently as a smart irrigation controller.  I have provided a copy of my ET calculations to them and I look forward to an integrated solution in a future release.

Kudos to Universal Devices for creating an open device that supports communication with external systems.  As a software developer this was one the key reasons that I selected the ISY.  On that note, I would love to see more capabilities added to read data from external services (SOAP, REST etc.) for direct use in programs.

This would be an very powerful feature that would allow the ISY to be much more responsive to external conditions (more than just WeatherBug, Flex Alert etc.). My current use of Flex Alert is a bit of a hack and is limited to polling a boolean value.  This limits the system to a fixed water balance approach.  The ability to retrieve a numerical value from a online resource would allow a lot more flexibility here.

Insteon

This has been my first foray into powerline communication (no X10 experience here) and I have been very impressed with the reliability of the system.  I had read some mixed reviews about Insteon reliability but I haven’t encountered any problems.  The system just works and I have had no dropped signals.

SimpleHomeNet EzFlora

The EzFlora works well in combination with the ISY-99i and it had been working perfectly until a few days ago when the device simply stopped working.  It died suddenly (errors starting showing up in the ISY event log) and would no longer respond to Insteon commands.

I tried multiple resets (using the process described on the SimpleHomeNet forums) but I had no luck getting it going again.  The status LED was no longer lit up when it was plugged in.  SmartHome customer service was very efficient and promptly issued an RMA to replace the faulty device.  I am hoping this is not a sign of poor quality control on SHN’s part but I will give them the benefit of the doubt for now until see how the replacement works out.

Categories: Home Automation Tags:

Building a Smart Irrigation Controller (Part 4)

August 19th, 2009 Comments off

Quick Links: Part 1, Part 2, Part 3, Part 4, Part 5

Software Components

As discussed in Part 3, there are two main software components that support the irrigation system: software that runs within the ISY and a web application that runs on an internet-hosted web server.

The ISY drives all of the scheduling and controls the physical triggering of the irrigation zone valves.  The web application performs the evapotranspiration (ET) calculations, tracks irrigation events (including rainfall) and tracks the water balance.

Note:  (8/6/2009) When I posted the first three parts of this series on the UDI Forums, Universal Devices expressed an interest in integrating evapotranspiration and water balance calculations into the ISY.  This is excellent news as this would allow the complete irrigation system to run independently without requiring a hosted web application component.  I am going to continue this series of posts based on my current system but I will update this later if move to a complete ISY-based system.  My current system is running very reliably so I will decide whether to switch later when this is released as part of the ISY firmware.

Web Application

The web application provides three main components: a water balance database, an HTTP interface that is used by programs in the ISY, and an online dashboard that displays the current status of the irrigation system.

Water Balance Database

The water balance is tracked in a single database table where each entry recorded in the table is a water balance event (i.e. any event that affects the water balance).

The water balance table tracks the following fields:

ID – A unique id for the event
TIMESTAMP – The timestamp of the event
EVENT – The event that occurred
AMOUNT – The amount of the event in inches

The current event types that are supported include:

ET – Evapotranspiration in inches
RF – Rainfall in inches
RO – Run-off in inches (used when the rainfall rate exceeds the infiltration rate of the soil)
IR – Irrigation amount in inches
ADJ – Adjustment in inches (used to adjust the water balance, for example to reset the water balance, while retaining the historical irrigation data)

HTTP Interface

The HTTP interface provides a mechanism for the ISY to communicate with the hosted web application.  The interfaces uses HTTP GET requests that pass parameters in the querystring (URL).  The interface supports the following request types:

processtodaysweather

Processes the weather for the current day.  This request is called once per day by the ISY (just before midnight).  This request triggers the web application to update the water balance with the current day’s evapotranspiration and rainfall.

The weather information that is required for these calculations is retrieved from WeatherBug (using the same station ID that is used in the climate module in the ISY).  Evapotranspiration (ET) is calculated using the FAO Penman-Monteith equation using the daily metrics provided by the WeatherBug station (min and max temperature, min and max humidity, and average windspeed).

The cumulative daily rainfall is provided directly by the WeatherBug station.

Example:

Request:

http://www.hostname.com/irrigation/control.php?action=processtodaysweather

Response (For diagnostic purposes):
SUCCESS: Recorded 0.25 inches of evapotranspiration.
SUCCESS: Recorded 0.05 inches of rainfall.

isirrigationreqd

This request type is different from the other request types in that it returns a boolean value to the ISY to determine if an irrigation cycle is required to be run.  The ISY currently does not have a way to read values from external systems so I am using the Flex Alert system to accomplish this.  This request type checks the water balance database and determines if the water balance deficit is greater than the amount specified.  If the amount is greater than the deficit an active Flex Alert response is provided, otherwise an inactive Flex Alert response is returned.  The Flex Alert status is then used in a program in the ISY to trigger an irrigation event.

Example:

Request:

http://www.hostname.com/irrigation/control.php?action=isirrigationreqd&amount=0.5

Response (Used by the Flex Alert system available in ISY 99 v2.7.3):
<?xml version=”1.0″ ?>
<flexalert>
<active>true</active>
</flexalert>

irrigated

Records an irrigation event.  This request is called by the ISY whenever an irrigation cycle is completed.  The amount of water that has been applied (in inches) is passed as a parameter.  The web application records this irrigation event in the water balance database.

Example:

Request:

http://www.hostname.com/irrigation/control.php?action=irrigation&amount=0.9

Response (For diagnostic purposes):
SUCCESS: Recorded 0.9 inches of irrigation.

Dashboard

I currently have a basic dashboard in place that shows the status of the irrigation system.  I plan to add more capabilities in the future (like the ability to change settings etc.) but the interface provides what I need for now.

The dashboard provides a summary of the recent activity and provides an interactive chart that displays the current months water balance events (daily evapotranspiration, cumulative water balance, irrigation and rainfall).

The screenshot below should be pretty self-explanatory.  Irrigation events are fairly evenly spaced in this example (about every 5 days) due to relatively consistent evapotranspiration and rainfall this month.

Irrigation Dashboard Screenshot

ISY Programs

The ISY drives the system by communicating with the web application and by running the actual irrigation cycles.  The system requires only three ISY programs and I will go over those in more detail here.

Irrigation Programs

Processing the Days Weather

As noted above, the web application provides a request type that allows the ISY to send a request to process the weather information for the current day.

The request is configured as a network resource in the networking module (named Irrigation.Resource.Action.ProcessTodaysWeather).

The following program calls this resource every night shortly before midnight.

Program Name: Irrigation.ProcessTodaysWeather

If
  Time is 11:45:00PM
Then
  Resource 'Irrigation.Resource.Action.ProcessTodaysWeather'
Else
  - No Actions - (To add one, press 'Action')

Checking if Irrigation is Required

As noted above, the Flex Alert system is used to inform the ISY when irrigation is required.   This is configured by setting the server URL in the Flex Alert Settings (Electricity > Main tab) to the “isirrigationreqd” request type.

Flex Alert Settings

Please note that the left-most portion of the URL is truncated in the screenshot due to the size of the text box.  The amount that is passed (0.6) is the allowable water deficit in inches.  If the water deficit is greater than the amount specified the Flex Alert response will return ‘true’ and an irrigation event will be run at the next scheduled time.

Running an Irrigation Cycle

The following program is responsible for checking the Flex Alert status and running the irrigation cycle:

Program Name: Irrigation.Run

If
  Time is Sunrise -  2 hours  and 30 minutes
  And Module 'Flex Alert' Status  is Active
  And Module 'Climate' Wind Speed &amp;lt;= 10 mph
  And Module 'Climate' Rain Today &amp;lt;= 0.1 &amp;quot;
Then
  Run Program 'Irrigation.Cycle' (Then Path)
  Resource 'Irrigation.Action.Irrigated'
Else
  - No Actions - (To add one, press 'Action')

The Flex Alert status is checked every morning at 2.5 hours before sunrise.  If the Flex Alert status is Active (irrigation is required) and there is no recent rainfall (that morning) and it is not windy then an irrigation cycle is run.

A separate program is used to run the actual irrigation cycle.  This program is designed to provide 0.9 inches of water to the lawn using 3 cycles of each of the zones (with a 50 minute soak time between cycles to reduce runoff due to the high clay content in our soil).

Program Name: Irrigation.Cycle

If
   - No Conditions - (To add one, press 'Schedule' or 'Condition')

Then
        Set 'Sprinkler Zone 1' On
        Wait  10 minutes
        Set 'Sprinkler Zone 3' On
        Wait  10 minutes
        Set 'Sprinkler Zone 2' On
        Wait  10 minutes
        Set 'Sprinkler Zone 4' On
        Wait  10 minutes
        Set 'Sprinkler Zone 4' Off
        Wait  50 minutes
        Set 'Sprinkler Zone 1' On
        Wait  10 minutes
        Set 'Sprinkler Zone 3' On
        Wait  10 minutes
        Set 'Sprinkler Zone 2' On
        Wait  10 minutes
        Set 'Sprinkler Zone 4' On
        Wait  10 minutes
        Set 'Sprinkler Zone 4' Off
        Wait  50 minutes
        Set 'Sprinkler Zone 1' On
        Wait  10 minutes
        Set 'Sprinkler Zone 3' On
        Wait  10 minutes
        Set 'Sprinkler Zone 2' On
        Wait  10 minutes
        Set 'Sprinkler Zone 4' On
        Wait  10 minutes
        Set 'Sprinkler Zone 4' Off
        Send Notification to All

Else
        Set 'Sprinkler Zone 1' Off

This program would be configured differently for different soil types with the objective of delivering a fixed amount of water while causing minimal runoff and minimal drainage below the root zone.

Clay has a high water capacity and releases water slowly so I provide a large amount of water during each irrigation cycle. If our soil was sandy a lesser amount of water would be applied. This would automatically result in a higher irrigation frequency with lesser amounts of water applied per irrigation cycle.

These amounts (allowable water deficit and irrigation amount) are set within the ISY so no changes are required in the online application to adjust the watering program.

Calibration of the irrigation amount can be done by using tuna cans or something similar to measure the watering rate of the system (inches per hour).  This can then be used to set the appropriate zone timings to acheive a specific irrigation amount (approximately 30 minutes per zone for my system).

Updating the Water Balance

When an irrigation cycle is run the Irrigation.Run program calls another network resource in the networking module (Irrigation.Resource.Action.Irrigation) to update the water balance with the irrigation amount.  This amount specified in the network resource is the amount water supplied by the irrigation cycle (0.9 inches in my case).

Continued (Read Part 5)

Categories: Home Automation Tags:

Building a Smart Irrigation Controller (Part 3)

August 1st, 2009 Comments off

Quick Links: Part 1, Part 2, Part 3, Part 4, Part 5

Software Design

I had reviewed the ISY-99i Wiki and the Forums in detail before purchasing so I had a general plan of action in mind for programming the system.

The main goals of the implementation were as follows:

  1. The ISY would control the system (i.e. all actions and events would be driven by the ISY)
  2. The ISY would connect to external resources but no external resources would connect to the ISY
  3. The irrigation system would run without requiring a home PC to be running

Modules Used

One of the great features of the ISY is its ability to integrate with other systems and services through add-on modules.

My system requires both the Climate Module and the Networking Module.

The Climate Module provides the ability to consume real-time weather information from a WeatherBug local weather station.  We have an elementary school less than a mile away that runs a WeatherBug station which was perfect for this project.  The weather reported from the school appears to track quite accurately with the actual conditions at our home.

The Networking Module provides the ability (among other things) to send requests to external resources including the ability to send HTTP GET and POST requests to any internet URL.  It should be noted that this capability allows you to send requests to external resources from programs but it does not allow you to use the response in programs.

Constraints and Design Decisions

The ISY programming model, while suitable for home automation, is primarily based on boolean logic and boolean conditions.  The software version that I am running (2.7.5 beta) does not support variables or calculations (although there is talk of adding these capabilities in a future release).

In either case, the level of variable and calculation support is not likely going to be suitable for handling the complexity of evapotranspiration (ET) calculations.

My solution was to build an externally hosted system to handle this side of the process.  The ISY would then integrate with this external system to determine when it was necessary to run an irrigation cycle.

I already had externally hosted web space that support PHP and MySQL so this requirement added no extra cost to the system.

Water Balance Irrigation Scheduling

My system uses the water balance approach to manage the frequency of irrigation events.

To keep things simple for now, I am using the Fixed Amount approach (as discussed in this paper) so that the irrigation cycle time can be kept constant.  The water balance is tracked on a daily basis and the lawn is irrigated only on days when the water deficit is sufficient to require an irrigation event.

Software Components

The ISY drives the irrigation scheduling but the tracking of the water balance is managed by a hosted application.  The diagram below shows the connectivity between the components of the system.

Network Diagram

In the next post in the series I will describe each of the software components in more detail but here is a high-level overview of how the system works:

  1. At the end of each day (shortly before midnight) the ISY sends an HTTP request to the online application to tell it capture the water balance metrics for the day (rainfall and calculated evapotranspiration).  This request causes the online application to download the day’s weather from WeatherBug (using the same address that would normally be used by the ISY).  The weather information is then used to calculate the evapotranspiration (ET) for the day.  The ET and the total rainfall for the day are then used to update the water balance in the database.
  2. The ISY polls the online application to determine if an irrigation cycle needs to be run based on the current water balance.  I ended up using the FlexAlert mechanism introduced in 2.7.3 to get this boolean value back into the ISY.  There was no other way that I could find to get data back into the ISY that could be used in programs – other than by proxying and manipulating the WeatherBug request/response but that didn’t turn out to be a very elegant solution.
  3. Before sunset every morning the ISY checks to see if the FlexAlert status is enabled.  If the status is enabled (and it isn’t windy or hasn’t rained significantly overnight) then an irrigation cycle is run.
  4. When the irrigation cycle is complete the ISY sends an HTTP request to the online application to tell it that a cycle has been completed and passes a parameter that specifies the amount of water that has been applied.  This causes the online application to update the water balance with the irrigation amount.

Continued (Read Part 4)

Categories: Home Automation Tags: