Skip navigation
All Places > PI Developers Club > Blog > 2015 > August
2015

Introduction

 

On one of my previous blog post Developing the Wikipedia Data Reference - Part 1, we have written the WikipediaWebService.cs class, which contains static methods to query wikipedia.

 

On this blog post, we will continue to work on this project in order to:

 

  • Create a custom data reference capable of querying wikipedia
  • Design an editor (Windows Form) in order to help the user set up the config string using PI System Explorer.
  • Automate the deployment process by setting up the pre-build and post-build events command line.
  • Test the data reference using PI System Explorer

 

Creating the Wikipedia Data Reference

 

We will continue working with the project WikipediaDR project available on part 1.  As there are a lot of examples about developing custom data references on PIDevClub, I will focus on the logic used in this project. You can download the source code package here.

 

First of all, let's define our goals:

 

  • The data reference should query wikipedia by providing an input string to the WikipediaWebServices.cs class.
  • The user should be able to choose if he wants to query the attribute element name, other attribute name from the same element, other attribute value or a custom text.
  • Due to performance issues, it is convenient to store the wikipedia description on memory so that the data reference does not need to make HTTP requests to Wikipedia whenever this data reference is accessed.
  • Wikipedia pages are dynamic. If you open the same page after one year, there is a high probability that the text would be changed. The data reference should query again wikipedia if the last query execution time was more than 6 months ago. The ConfigString will store the last query execution time in order to implement this logic.

 

The code snippet from the Wikipedia.cs is below:

 

using System;
using System.ComponentModel;
using System.Globalization;
using System.Runtime.InteropServices;
using System.Text;
using System.Windows.Forms;
using OSIsoft.AF.Asset;
using OSIsoft.AF.UnitsOfMeasure;
using OSIsoft.AF.Time;




namespace OSIsoft.AF.Asset.DataReference.WikipediaDR
{
    [Serializable]
    [Guid("B1D3CF85-D3CF-4033-A058-7554BE3EEEFF")]
    [Description("Wikipedia; Get description of a word from the Wikipedia.")]
    public class Wikipedia : AFDataReference
    {
        private Tuple<string, string, AFTime> _lastQueriedValue;


        private string _configString;


        public enum SourceOption
        {
            ElementName, AttributeName, AttributeValue, Custom
        }


        public Wikipedia()
            : base()
        {
        }


        public override Type EditorType
        {
            get { return typeof(ConfigStringEditor); }
        }


        public override AFDataReferenceContext SupportedContexts
        {
            get
            {
                AFDataReferenceContext supportedContexts = AFDataReferenceContext.All;
                return supportedContexts;
            }
        }


        public override AFDataReferenceMethod SupportedMethods
        {
            get
            {
                AFDataReferenceMethod supportedMethods = AFDataReferenceMethod.GetValue;
                return supportedMethods;
            }
        }


        public override AFValue GetValue(object context, object timeContext, AFAttributeList inputAttributes, AFValues inputValues)
        {
            if (string.IsNullOrEmpty(ConfigString))
                throw new Exception("ConfigString cannot be null or empty");


            if (Attribute == null || (Attribute.Type != typeof(string) && Attribute.Type != typeof(Object)))
                throw new Exception("The attribute type must be a String");


            AFTime timeStamp;
            if (timeContext is AFTime)
                timeStamp = (AFTime)timeContext;
            else if (timeContext is AFTimeRange)
                timeStamp = ((AFTimeRange)timeContext).EndTime;
            else
                timeStamp = AFTime.Now;


            var lastQueriedValue = _lastQueriedValue; // save queryString, result, query timestamp
            string wikipediaQuery = GetQueryString(context, timeContext);
            if (lastQueriedValue != null && lastQueriedValue.Item1 == wikipediaQuery && (AFTime.Now - lastQueriedValue.Item3).TotalDays < 180)
                return new AFValue(lastQueriedValue.Item2, timeStamp);


            string queryResult = WikipediaWebService.GetWikipediaDescription(wikipediaQuery).Trim().Replace(";", ",");
            if (queryResult.Contains("Error: "))
                return new AFValue(queryResult, timeStamp, null, AFValueStatus.Bad);


            lastQueriedValue = Tuple.Create(wikipediaQuery, queryResult, AFTime.Now); // keep track of timestamp when source was queried
            _lastQueriedValue = lastQueriedValue; // cache the last result


            return new AFValue(queryResult, timeStamp);
        }


        private string GetQueryString(object context, object timeContext)
        {
            string substitutedConfig = SubstituteParameters(_configString, this, context, timeContext).Trim();
            if (substitutedConfig.StartsWith("%@") && substitutedConfig.EndsWith("%"))
            {
                string attributeName = substitutedConfig.Substring(2, substitutedConfig.Length - 3);
                var referencedAttribute = base.GetAttribute(attributeName);
                if (!ReferenceEquals(referencedAttribute, null))
                {
                    var value = referencedAttribute.GetValue(context, timeContext, null);
                    if (value != null && value.Value != null)
                    {
                        return value.Value.ToString();
                    }
                }
            }


            return substitutedConfig;
        }


        private bool CheckIfAttributeExists(string attributeName)
        {
            if (this.Attribute.Element.Attributes[attributeName] == null)
                return false;
            else
                return true;
        }




        public void SaveConfigString()
        {
            base.SaveConfigChanges();
        }


        public override string ConfigString
        {
            get
            {
                return _configString;
            }


            set
            {
                if (ConfigString != value)
                {
                    _configString = value;
                    SaveConfigChanges();
                }
            }
        }
    }
}

 

Note that the AFDataReference.SubstituteParameters method is used which means that if the config string is %Element%, this method will convert it to the element name. If the config string is %@AttributeName%, it will search on wikipedia for the value of the attribute whose name is AttributeName.

Designing the Editor

 

An editor helps a lot when the user wants to set up the ConfigString using PI System Explorer. We've created the ConfigStringEditor.cs Windows Form class that looks like the window below:

 

 

 

Right click on the ConfigStringEditor.cs and click on "View Code...." and add the following code snippet:

 

using OSIsoft.AF.Asset.DataReference;
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;


namespace OSIsoft.AF.Asset.DataReference.WikipediaDR
{
    public partial class ConfigStringEditor : Form
    {
        private Wikipedia dataReference = null;
        private List<string> attributeNames = new List<string>();
        public ConfigStringEditor(Wikipedia dataReference, bool bReadOnly)
            : base()
        {
            InitializeComponent();
            this.dataReference = dataReference;
            cbDrOptions.SelectedItem = "Element name";




            cbAttributes.Items.Clear();
            if (dataReference.Attribute != null)
            {
                attributeNames = GetAttributeNames((AFElement)this.dataReference.Attribute.Element);
                cbAttributes.Items.AddRange(attributeNames.ToArray());
                if (attributeNames.Count > 0)
                {
                    cbAttributes.SelectedItem = attributeNames[0];
                }
            }
            cbDrOptions_SelectedIndexChanged(null, null);
            LoadValuesFromConfigString();
        }


        private List<string> GetAttributeNames(AFElement parentElement)
        {
            List<string> attributeNames = new List<string>();


            foreach (AFAttribute attribute in parentElement.Attributes)
            {
                if (attribute.Name != this.dataReference.Attribute.Name)
                {
                    attributeNames.Add(attribute.Name);
                }
            }
            return attributeNames;


        }


        private void cbDrOptions_SelectedIndexChanged(object sender, EventArgs e)
        {


            if (cbDrOptions.SelectedItem.ToString() == "Element name")
            {
                lbCustom.Visible = false;
                lbCustom.Text = "";
                tbCustom.Visible = false;
                cbAttributes.Visible = false;
            }
            else if (cbDrOptions.SelectedItem.ToString() == "Attribute name")
            {
                lbCustom.Visible = true;
                lbCustom.Text = "Select attribute name option:";
                tbCustom.Visible = false;
                cbAttributes.Visible = true;
                if (attributeNames.Count > 0)
                {
                    cbAttributes.SelectedItem = attributeNames[0];
                }


            }
            else if (cbDrOptions.SelectedItem.ToString() == "Attribute value")
            {
                lbCustom.Visible = true;
                lbCustom.Text = "Select attribute name value:";
                tbCustom.Visible = false;
                cbAttributes.Visible = true;
                if (attributeNames.Count > 0)
                {
                    cbAttributes.SelectedItem = attributeNames[0];
                }
            }
            else if (cbDrOptions.SelectedItem.ToString() == "Custom")
            {
                lbCustom.Visible = true;
                lbCustom.Text = "Write custom words:";
                tbCustom.Visible = true;
                cbAttributes.Visible = false;


            }




        }




        private void btnOK_Click(object sender, EventArgs e)
        {
            try
            {
                if (cbDrOptions.SelectedItem.ToString() == "Element name")
                {
                    dataReference.ConfigString = "%Element%";
                }
                else if (cbDrOptions.SelectedItem.ToString() == "Attribute name")
                {
                    dataReference.ConfigString = cbAttributes.SelectedItem.ToString();
                }
                else if (cbDrOptions.SelectedItem.ToString() == "Attribute value")
                {
                    dataReference.ConfigString = "%@" + cbAttributes.SelectedItem.ToString() + "%";
                }
                else if (cbDrOptions.SelectedItem.ToString() == "Custom")
                {
                    dataReference.ConfigString = tbCustom.Text;
                    cbAttributes.SelectedItem = "";
                }
                else
                {
                    throw new ArgumentException();
                }
                dataReference.SaveConfigString();
                this.Close();
            }
            catch (Exception ex)
            {
                DialogResult = DialogResult.None;
                MessageBox.Show("There was an error saving the ConfigString values. Error: " + ex.Message);
            }
            finally
            {


            }
        }


        private string GetAttributeNameFromConfigString()
        {
            foreach (var item in cbAttributes.Items)
            {
                if ((string.IsNullOrEmpty(dataReference.ConfigString) == false) && (dataReference.ConfigString.Contains(item.ToString())))
                {
                    return item.ToString();
                }
            }
            return null;
        }


        private void LoadValuesFromConfigString()
        {
            string attributeName = GetAttributeNameFromConfigString();
            try
            {


                if (dataReference.ConfigString == "%Element%")
                {
                    cbDrOptions.SelectedItem = "Element name";
                }
                else if (string.IsNullOrEmpty(attributeName) == false)
                {
                    if (dataReference.ConfigString.Contains("@") == true)
                    {
                        cbDrOptions.SelectedItem = "Attribute value";
                        cbAttributes.SelectedItem = attributeName;
                    }
                    else
                    {
                        cbDrOptions.SelectedItem = "Attribute name";
                        cbAttributes.SelectedItem = attributeName;
                    }
                }
                else
                {
                    cbDrOptions.SelectedItem = "Custom";
                    tbCustom.Text = dataReference.ConfigString;
                }
            }
            catch (Exception ex)
            {
                MessageBox.Show("There was an error loading the values from Config String. Error: " + ex.Message);
            }
        }




        private void btnCancel_Click(object sender, EventArgs e)
        {
            this.Close();
        }
    }
}

 

 

 

Here is a summary of what the code snippet above does:

 

  • Open and display the form.
  • Load the other attributes from the same element.
  • Update the UI according to the selected combo box option.
  • Set up the data reference ConfigString when the user press the "OK" button.

 

Automating the deployment process

 

Whenever you are developing a custom data reference, there are a lot of steps that you need to follow in order to update and debug the new version of the custom assembly by attaching to process. Here they are:

 

  • Close PI System Explorer.
  • Delete the .dll and .pdb files from the C:\ProgramData in case you are using the same version on the assembly information but with an updated code.
  • Register/Update the new assembly using regplugin.
  • Open PI System Explorer.
  • Copy the .pdb file to the proper C:\ProgramData subfolder in order to be able to attach Visual Studio Debugger to the AFExplorer.exe process. The .dll file will be copied by PI System Explorer automatically.

 

You can automate this process by setting up pre-build and post-build events command line on your project settings in Visual Studio.

 

Pre-build event command line:

 

tasklist /fi "imagename eq afexplorer.exe" |find ":" > nul
if errorlevel 1 taskkill /IM afexplorer.exe
"%pihome%\AF\regplugin" -u '$(TargetFileName)'
SET dllFile="C:\ProgramData\OSIsoft\AF\PlugIns\1.1.0.0\4.0\$(TargetFileName)"
SET pdbFile="C:\ProgramData\OSIsoft\AF\PlugIns\1.1.0.0\4.0\$(TargetName).pdb"
IF EXIST %dllFile% del /F %dllFile%
IF EXIST %pdbFile% del /F %pdbFile%

 

Post-build event command line:

 

"%pihome%\AF\regplugin" "$(TargetPath)" "$(TargetDir)HtmlAgilityPack.dll" "$(TargetDir)Newtonsoft.Json.dll" /own:WikipediaDR.dll
copy "$(TargetDir)$(TargetName).pdb" "C:\ProgramData\OSIsoft\AF\PlugIns\1.1.0.0\4.0"
"%pihome%\AF\afexplorer.exe"

 

Please refer to the screenshot below if you don't know exactly where you need to set up those settings on your project properties.

 

 

Some comments of this automated deployment process:

 

  • Tasklist is used to close PI System Explorer in case it is opened.
  • The lines above where written on Windows 7 - 32 bits. You might want to change %PIHOME% to %PIHOME64% according to your development system. The same might be needed for regplugin as you might prefer using regplugin64.
  • The other libraries (HtmlAgilityPack.dll and Newtonsoft.Json.dll) need to be referenced when the WikipediaDR library is registered as without them, WikipediaDR will not work properly.
  • We have taken advantage of using the macros that Visual Studio provides. Note that under the hood, "$(TargetName) is changed to "WikipediaDR". Visual Studio provides a great window to help you write the command lines as shown below.

 

 

Testing the assembly

 

Now that we have finished developing and deploying the assembly, let's test it:

 

  • Open PI System Explorer and create a new database WikiDb
  • Create an element called Brazil and create two attributes: "Uruguay" and "Wikipedia Description" on the Brazil element.
  • The value of the attribute Uruguay should be Argentina.

 

The screenshot below shows how your AF hierarchy should looks like.

 

 

Select the Wikipedia Description attribute and make sure you are able to get the description from Brazil, Uruguay and Argentina by changing the ConfigString through the editor. Finally, get the description from "OSIsoft" by changing the source option to "Custom" under the Wikipedia ConfigString Editor. The result is shown below:

Conclusion

 

There are many use cases where the Wikipedia Data Reference would be a good fit. We have developed a sample application that shows weather information including forecast data from worldwide cities. The wikipedia data reference was used to show the description from each city. Make sure you comply with Wikipedia terms of service when using this data reference in your applications.

 

I hope you have found this blog post interesting and stay tuned for the upcoming ones!

Introduction

 

During the preparation for the Programming Hackathon, we had to learn how to manage not only the security of each domain user account programmatically, but also new security model from PI AF.

 

Our goal was to create 250 domain users and 50 domain groups as each group had 5 members/users. Each hackathon group received an AF database in order to be able to manipulate any AF object. Only its members could view/create/edit/delete its content. The other group members couldn't view the database content.

 

The same happened for the PI Data Archive. Each group had their own PI Points while the other group members couldn't realize their existence. All hackathon participants were able to create PI Points on a single central PI Data Archive.

 

Finally, each group was able to share its data with another specific group in case their members decide to do it.  Those were the restrictions we wanted to implement programmatically. How we were able to do that?

 

Managing user and group objects from the domain

 

There are a couple of approaches to solve this problem but we've decided to use Windows PowerShell. Our code snippet creates AD objects accessing the data from some spreasheets containing the AD users and groups.

 

 

The first spreadsheet is called NetHackGroups.csv. It has the following structure to create groups:

 

GroupName
Group01
Group02
Group03
Group04
Group05

 

The powershell code snippet below creates the requested AD groups:

 

cls;
Import-Module ActiveDirectory
New-ADGroup -Name "All Hackathon Users" -Path "CN=Users,DC=osiproghack,DC=net" -Description "All Hackathon Users"  -GroupCategory Security -GroupScope Universal
Import-Csv ".\NewHackGroups.csv" | ForEach-Object {
Write-Host $_.GroupName
$groupName = "Hackathon" + $_.GroupName
$groupDescription =  $_.GroupName + " from the Programming Hackathon"
Write-Host $groupName $groupDescription
New-ADGroup -Name $groupName  -Path "CN=Users,DC=osiproghack,DC=net" -Description $groupDescription  -GroupCategory Security -GroupScope Universal 
}

 

In order to use the AD command-lets, we need to import the Active directory module. Then a group called "All Hackathon Users" is created so that all hackathon users will become members. Later, we import the spreadsheet and process each line which will be responsable for creating a group with a description on a specific AD folder (CN = Users).

 

The second spreadsheet is called NetHackUsers.csv. It has the following structure to create users and it assigns each user to its group:

 

Name,samAccountName,ParentOU,Password,Group
Hackathon User 001,hackuser001,"CN=Users,DC=osiproghack,DC=net",8M5s6Kd1,HackathonGroup01
Hackathon User 002,hackuser002,"CN=Users,DC=osiproghack,DC=net",ZmXM0NJ2,HackathonGroup01
Hackathon User 003,hackuser003,"CN=Users,DC=osiproghack,DC=net",zCyruR02,HackathonGroup01
Hackathon User 004,hackuser004,"CN=Users,DC=osiproghack,DC=net",PcQLnUg2,HackathonGroup01
Hackathon User 005,hackuser005,"CN=Users,DC=osiproghack,DC=net",q2rphYM2,HackathonGroup01
Hackathon User 006,hackuser006,"CN=Users,DC=osiproghack,DC=net",GSJJbb32,HackathonGroup02
Hackathon User 007,hackuser007,"CN=Users,DC=osiproghack,DC=net",hsknUfj2,HackathonGroup02
Hackathon User 008,hackuser008,"CN=Users,DC=osiproghack,DC=net",7IBHOiQ2,HackathonGroup02
Hackathon User 009,hackuser009,"CN=Users,DC=osiproghack,DC=net",XidlIm62,HackathonGroup02
Hackathon User 010,hackuser010,"CN=Users,DC=osiproghack,DC=net",y84FBpn2,HackathonGroup02
Hackathon User 011,hackuser011,"CN=Users,DC=osiproghack,DC=net",OYWj5tT2,HackathonGroup03
Hackathon User 012,hackuser012,"CN=Users,DC=osiproghack,DC=net",oyxEzw92,HackathonGroup03
Hackathon User 013,hackuser013,"CN=Users,DC=osiproghack,DC=net",FOPis0q2,HackathonGroup03
Hackathon User 014,hackuser014,"CN=Users,DC=osiproghack,DC=net",foqCm3W2,HackathonGroup03
Hackathon User 015,hackuser015,"CN=Users,DC=osiproghack,DC=net",5EIgg7D2,HackathonGroup03
Hackathon User 016,hackuser016,"CN=Users,DC=osiproghack,DC=net",WejAZAt2,HackathonGroup04
Hackathon User 017,hackuser017,"CN=Users,DC=osiproghack,DC=net",w3BeTEa2,HackathonGroup04
Hackathon User 018,hackuser018,"CN=Users,DC=osiproghack,DC=net",NTc8NHG2,HackathonGroup04
Hackathon User 019,hackuser019,"CN=Users,DC=osiproghack,DC=net",nt4cGLx2,HackathonGroup04
Hackathon User 020,hackuser020,"CN=Users,DC=osiproghack,DC=net",DJV6AOd2,HackathonGroup04
Hackathon User 021,hackuser021,"CN=Users,DC=osiproghack,DC=net",ejwb4SJ2,HackathonGroup05
Hackathon User 022,hackuser022,"CN=Users,DC=osiproghack,DC=net",49O5xV02,HackathonGroup05
Hackathon User 023,hackuser023,"CN=Users,DC=osiproghack,DC=net",UZpZrZg2,HackathonGroup05
Hackathon User 024,hackuser024,"CN=Users,DC=osiproghack,DC=net",vzH3lcN2,HackathonGroup05

 

 

Let's describe each column:

 

  • Name: the display name of the user.
  • SamAccountName: the account logon name or the user object is stored - in fact the legacy NetBIOS form as used in the naming notation.
  • ParentOU: specifies where the object will be created on the AD folders hierarchy.
  • Password: the user password.
  • Group: the name of the group which the created user will be member of.

 

 

Finally, the PowerShell code snippet below creates the users using the data from the spreadsheet and it assigns each user not only to a domain group but also to the group "All Hackathon Users" .

 

Import-Csv ".\NewHackUsers.csv" | ForEach-Object {
Write-Host ""
Write-Host $_.Name
Write-Host $_."samAccountName";
Write-Host $_."Password";
Write-Host $_."Group";


 $userPrincinpal = $_."samAccountName" + "@osiproghack.net"
New-ADUser -Name $_.Name `
 -Path $_."ParentOU" `
 -SamAccountName  $_."samAccountName" `
 -UserPrincipalName  $userPrincinpal `
 -AccountPassword (ConvertTo-SecureString $_."Password" -AsPlainText -Force) `
 -ChangePasswordAtLogon $false  `
 -Enabled $true


 Add-ADGroupMember $_."Group" $_."samAccountName";
Add-ADGroupMember "All Hackathon Users"  $_."samAccountName";
}

 

Managing the PI System security programmatically

 

The new security model from PI AF 2.7 introduces a similar model compared to PI Data Archive using mapping and identities as described on PI AF 2015 Release Notes:

 

"An update to the security model to make it similar to the model exposed by the PI Data Archive.  ACL’s on AF Objects are no longer being tied directly to Windows Principals, but are instead defined using AF defined identities."

 

 

Creating identities and mappings on the PI Data Archive

 

As we will have to create identities and mappings programmatically on the PI Data Archive and the PI AF Server, let's start with the first option for you to remember how this is done. Please refer to the code snippet below:

 

  private static void CreatePIServerSecurityObjs()
        {
            PIServer piServer = new PIServers()["MERCURY001"];
            piServer.Connect();

            for (int i = 1; i < 51; i++)
            {
                int k = 100 + i;
                string name = "HackathonGroup" + k.ToString().Substring(1, 2);
                if (piServer.Identities[name] == null)
                {
                    PIIdentity newIdentity = piServer.Identities.Add(name, PIIdentityType.PIIdentity);
                    newIdentity.AllowExplicitLogin = false;
                    newIdentity.AllowTrusts = false;
                    newIdentity.Description = "Identity for group " + name;
                    newIdentity.IsEnabled = true;
                    newIdentity.CheckIn();
                }

                if (piServer.IdentityMappings[name] == null)
                {
                    PIIdentityMapping newMapping = piServer.IdentityMappings.Add(name, @"OSIPROGHACK\" + name, name);
                    newMapping.Description = "Mapping for group " + name;
                    newMapping.CheckIn();
                }
            }
        }

 

First we connect to the PI Data Archive and foreach Hackathon group the program will:

 

  • Generate the name for the identity and mapping which in this case would be the same. The name for the first group would be HackathonGroup01. It was needed to add 100 to integer i before getting the last two digits in order to generate the string name  HackathonGroup01 and not HackathonGroup1.
  • With this string, the program checks if there is an identity with the generated name. If it does not exist, it will create one with proper description. The method CheckIn() needs to be called in order to save the changes.
  • A similar process happens with the mapping (PIIdentityMapping class).

 

Running the method above, it creates one identity and one mapping per group. The upcoming PI Points created by a group needs to have their security settings edited by its members (programmaticaly or using PI SMT) so that the other 49 groups won't have access to them.

 

Creating the AF databases

 

Now that the security of the PI Data Archive was handled, let's start creating our 50 databases on PI AF Server. Each database needs to be managed only by a single group. As you already know, it is pretty easy to create 50 databases using PI AF SDK:

 

        private static void CreateDbs()
        {
            PISystem piSystem = new PISystems()["MERCURY001"];
            piSystem.Connect();


            for (int i = 1; i < 51; i++)
            {
                int k = 100 + i;
                string name = "DatabaseHackGroup" + k.ToString().Substring(1, 2);
                AFDatabase db = piSystem.Databases[name];
                if (db == null)
                {
                    piSystem.Databases.Add(name);
                }
            }
            piSystem.CheckIn();


        }

 

 

 

Creating identities and mappings on the new PI AF Server 2.7

 

The code snippet for creating identities and mapping on PI AF Server is very similar compared to the PI Data Archive version:

 


        private static void CreatePISystemSecurityObjs()
        {
            PISystem piSystem = new PISystems()["MERCURY001"];
            piSystem.Connect();


            for (int i = 1; i < 51; i++)
            {
                int k = 100 + i;
                string name = "HackathonGroup" + k.ToString().Substring(1, 2);
                AFSecurityIdentity newIdentity = piSystem.SecurityIdentities[name];
                if (newIdentity == null)
                {
                    newIdentity = piSystem.SecurityIdentities.Add(name);
                    newIdentity.Description = "Identity for group " + name;
                    newIdentity.IsEnabled = true;
                    newIdentity.CheckIn();
                }


                AFSecurityMapping newMapping = piSystem.SecurityMappings[name];
                if (newMapping == null)
                {
                    System.Security.Principal.NTAccount account = new System.Security.Principal.NTAccount("OSIPROGHACK", name);
                    newMapping = piSystem.SecurityMappings.Add(name, account, newIdentity);
                    newMapping.Description = "Mapping for group " + name;
                    newMapping.CheckIn();
                }          
            }


            AFSecurityIdentity newIdentityForAll = piSystem.SecurityIdentities["AllHackUsers"];
            if (newIdentityForAll == null)
            {
                newIdentityForAll = piSystem.SecurityIdentities.Add("AllHackUsers");
                newIdentityForAll.Description = "Identity for all hackathon users";
                newIdentityForAll.IsEnabled = true;
                newIdentityForAll.CheckIn();
            }


            AFSecurityMapping newMappingForAll = piSystem.SecurityMappings["AllHackUsers"];
            if (newMappingForAll == null)
            {
                System.Security.Principal.NTAccount account = new System.Security.Principal.NTAccount("OSIPROGHACK", "All Hackathon Users");
                newMappingForAll = piSystem.SecurityMappings.Add("AllHackUsers", account, newIdentityForAll);
                newMappingForAll.Description = "Mapping for all hackathon users";
                newMappingForAll.CheckIn();
            }
        }

 

Here are the main differences between them:

 

  • When calling the method SecurityMapping.Add(), one of the inputs is an System.Security.Principal.NTAccount object whereas on the PI Data Archive version, this account was referred as a string.
  • Besides the creation of the 50 identities and 50 mappings, we have created a mapping and an identity for all hackathon users group.

 

Setting up proper security on the PI AF Server

 

Finally, the method SetPISystemSecurityObjs() will set up the PI System and AF databases security property in order to apply the desired restrictions. Please refer to the code snippet below:

 

 

        private static void SetPISystemSecurityObjs()
        {
            PISystem piSystem = new PISystems()["MERCURY001"];
            piSystem.Connect();
            AFSecurity security1 = piSystem.GetSecurity(AFSecurityItem.Default);
            security1.SetSecurityString(security1.GetSecurityString() + "|AllHackUsers:A(r,rd)", true);


            AFSecurity security2 = piSystem.GetSecurity(AFSecurityItem.SecurityIdentity);
            security2.SetSecurityString(security2.GetSecurityString() + "|AllHackUsers:A(r,rd)", true);


            AFSecurity security3 = piSystem.GetSecurity(AFSecurityItem.SecurityMapping);
            security3.SetSecurityString(security3.GetSecurityString() + "|AllHackUsers:A(r,rd)", true);




            for (int i = 1; i < 51; i++)
            {
                Console.WriteLine("Processing item " + i.ToString());
                AFDatabase groupDb = piSystem.Databases[GetDatabaseName(i)];
                IList<AFSecurity> groupSecurityObjs = new List<AFSecurity>();
                foreach (AFSecurityItem securityItem in Enum.GetValues(typeof(AFSecurityItem)))
                {
                    AFSecurity groupSecurityObj = groupDb.GetSecurity(securityItem);
                    if (groupSecurityObj != null)
                    {
                        groupSecurityObjs.Add(groupSecurityObj);
                    }
                }
                AFSecurity.AddIdentity(piSystem, piSystem.SecurityIdentities[GetIdentityName(i)], groupSecurityObjs, AFSecurityRights.All, AFSecurityRights.None, AFSecurityOperation.Merge, true);
            }
        }


        private static string GetDatabaseName(int i)
        {
            int k = 100 + i;
            string name = "DatabaseHackGroup" + k.ToString().Substring(1, 2);
            return name;
        }


        private static string GetIdentityName(int i)
        {
            int k = 100 + i;
            string name = "HackathonGroup" + k.ToString().Substring(1, 2);
            return name;
        }

 

The PI System and the 50 AF databases need to have their security settings changed.

 

Concerning the PI System, calling piSystem.GetSecurity(AFSecurityItem) will return an AFSecurity object. For each type of AFSecurityItem, you will edit the security of a different aspect of this object security. If you want to change the security mappings from the selected PI System, you shall call piSystem.GetSecurity(AFSecurityItem.SecurityMapping). The returned AFSecurity has a method called SetSecurityString() which lets you change the security string. You can call GetSecurityString() in order to view the current security string of the object before changing it.

 

Concerning each one of the AF Databases, we will get the group database and store it on groupDb variable. For each type of AFSecurityItem we will call groupDb.GetSecurity(AFSecuirtyItem) whose returned AFSecurity will be added to an IList<AFSecurity> groupSecurityObjs. Finally, AFSecurity has a static method called AddIdentity which is used to set up its security settings from each database with the groupSecurityObjs and its respective identity.

 

 

 

Conclusions

 

There are some scenarios where you need to manage a large number of users on your PI System. As the recommended security of the PI System is Windows Integrated Security which uses objects from the Active Directory, creating objects on the domain programmatically is a very interesting skill to have. After doing this procedure, PI AF SDK provides enough methods to manage the mappings and identities from the PI Data Archive and from PI AF Server in order to secure your environment properly.

 

Handling the security manually using PI System Explorer may lead to security breach caused by humans' mistakes. A safer option is to manage the security programmatically as for well tested code, security breaches are generally harder to find.

 

Hope you have found this blog post interesting and I would like to thank you for reading it!

As I was trying to write a C# application that uses PI Web API and looking up various ways to make a REST call in .NET applications, I came across the following methods to consume REST APIs in .NET:

  • WebClient
  • HttpWebRequest
  • HttpClient
  • Other libraries (e.g. RestSharp)

 

It looks like the new API that Microsoft offers, HttpClient (namespace: System.Net.Http), provides some powerful functionality with better syntax support for newer threading features. For more information about the differences of various ways to make REST calls, please refer to the following resources:

 

Looking through the pros and cons of each, I have decided to give HttpClient a try. In the following blog post, we will be writing a wrapper around HttpClient for easy access to PI Web API. Note that this is not a comprehensive HttpClient wrapper for PI Web API, the idea is to share some development thoughts while testing the class out. If you need a quick and easy way to access PI Web API in your development project, you can use this class and focus on the data access logic.

 

The following example is created in Visual Studio 2013, .NET 4.5 and tested with PI Web API 2015 R2.

 

 

Getting Started

First, we will create a wrapper. Start a new Visual Studio project and name the class PIWebAPIClient.

 

namespace piwebapi_cs_helper
{
    public class PIWebAPIClient
    {
    }
}

 

Since we will be handling JSON objects, let’s add the Newtonsoft.Json package from the package manager console:

Install-Package Newtonsoft.Json

 

And add the using directive:

using Newtonsoft.Json.Linq;

 

Let’s also add a reference to System.Net.Http and add the using directive since we are testing the HttpClient class!

using System.Net.Http

 

 

Creating and Disposing the HttpClient

In the following, we will write constructors and methods to initialize a HttpClient intended to be used to make multiple HTTP requests.

 

        private HttpClient client;

        /* Initiating HttpClient using the default credentials.
         * This can be used with Kerberos authentication for PI Web API. */
        public PIWebAPIClient()
        {
            client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true });
        }

        /* Initializing HttpClient by providing a username and password. The basic authentication header is added to the HttpClient.
         * This can be used with Basic authentication for PI Web API. */
        public PIWebAPIClient(string userName, string password)
        {
            client = new HttpClient();
            string authInfo = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(String.Format("{0}:{1}", userName, password)));
            client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", authInfo);
        }

        /* Disposing the HttpClient. */
        public void Dispose()
        {
            client.Dispose();
        }

 

As a simple example, we did not add any additional headers for the HttpClient. Feel free to add headers appropriate for your application. In addition, although HttpClient does implement the IDisposable interface, many MSDN examples did not explicitly call Dispose(). We will include it for completion sake.

 

 

Asynchronous GET and POST request

We are ready to make HTTP requests! Since PI Web API only returns JSON objects by default, we will return a NewtonSoft JObject from our async GET method. For POST, we will be accepting a string payload in JSON format. If the response message indicates that the request is not successful, a HttpRequestException will be thrown with the response message.

 

        /* Async GET request. This method makes a HTTP GET request to the uri provided
         * and throws an exception if the response does not indicate a success. */
        public async Task<JObject> GetAsync(string uri)
        {
            HttpResponseMessage response = await client.GetAsync(uri);
            string content = await response.Content.ReadAsStringAsync();
            if (!response.IsSuccessStatusCode)
            {
                var responseMessage = "Response status code does not indicate success: " + (int)response.StatusCode + " (" + response.StatusCode + " ). ";
                throw new HttpRequestException(responseMessage + Environment.NewLine + content);
            }
            return JObject.Parse(content);
        }

        /* Async POST request. This method makes a HTTP POST request to the uri provided
         * and throws an exception if the response does not indicate a success. */
        public async Task PostAsync(string uri, string data)
        {
            HttpResponseMessage response = await client.PostAsync(uri, new StringContent(data, Encoding.UTF8, "application/json"));
            string content = await response.Content.ReadAsStringAsync();
            if (!response.IsSuccessStatusCode)
            {
                var responseMessage = "Response status code does not indicate success: " + (int)response.StatusCode + " (" + response.StatusCode + " ). ";
                throw new HttpRequestException(responseMessage + Environment.NewLine + content);
            }
        }

 

At times, you might need to submit other HTTP requests to PI Web API (e.g. PUT, PATCH). While you can use HttpClient.PutAsync for a PUT request, it doesn’t have a method to support PATCH request out-of-the-box. If you are in a situation to make a PATCH call, there are many online examples to do so. For more information, refer to this previous discussion.

 

 

(Optional) Additional methods to make Synchronous Calls (e.g. Console Application)

If you are writing a simple console application and would like to synchronously call the GetAsync/PostAsync method, you can add the following methods:

 

        /* Calling the GetAsync method and waiting for the results. */
        public JObject GetRequest(string url)
        {
            Task<JObject> t = this.GetAsync(url);
            t.Wait();
            return t.Result;
        }

        /* Calling the PostAsync method and waiting for the results. */
        public void PostRequest(string url, string data)
        {
            Task t = this.PostAsync(url, data);
            t.Wait();
        }

 

 

Testing synchronous calls in a Console application

Let’s build the solution and test!

 

First, let’s add references to our PIWebAPIClient helper, as well as to Newtonsoft.Json.

using piwebapi_cs_helper;
using Newtonsoft.Json.Linq;

 

and write a simple console application. The following console application accepts a URL (i.e. the PI Web API REST endpoint) and prints out the response message. Note that we are using the PIWebAPIClient constructor that uses the default credentials. This method works when your PI Web API instance is set up using Kerberos.

namespace piwebapi_cs_console_test
{
    class Program
    {
        /* Console application that makes GET request to a specified URL and display the response
         * as a string to the console. */
        static void Main(string[] args)
        {
            PIWebAPIClient piWebAPIClient = new PIWebAPIClient();
            do
            {
                try
                {
                    Console.Write("Enter URL: ");
                    string url = Console.ReadLine();
                    JObject jobj = piWebAPIClient.GetRequest(url);
                    Console.WriteLine(jobj.ToString());
                }
                catch (AggregateException ex)
                {
                    foreach (var e in ex.InnerExceptions)
                    {
                        Console.WriteLine(e.Message);
                    }
                }
                finally
                {
                    Console.WriteLine("Press any key to continue (esc to exit)...");
                }

            } while (Console.ReadKey().Key != ConsoleKey.Escape);
            piWebAPIClient.Dispose();
            Console.ReadKey();
        }
    }
}

 

 

As you can see, we can make simple synchronous requests!

 

 

Testing asynchronous calls in a WPF application

Next, we will write a WPF application to test making asynchronous GET and POST requests. Again, add references to the following:

using piwebapi_cs_helper;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Net.Http;

 

If you are interested, the WPF xaml configuration is as follow:

<Window x:Class="piwebapi_cs_wpf_test.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="325" Width="500">
    <Grid Margin="0,0,0,11">
        <Grid.RowDefinitions>
            <RowDefinition Height="30" />
            <RowDefinition Height="30" />
            <RowDefinition Height="30" />
            <RowDefinition Height="30" />
            <RowDefinition Height="30" />
            <RowDefinition Height="Auto" />
        </Grid.RowDefinitions>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="100" />
            <ColumnDefinition Width="300" />
        </Grid.ColumnDefinitions>
        <Label Grid.Row="0" Grid.Column="0" Content="User name" />
        <Label Grid.Row="1" Grid.Column="0" Content="Password" />
        <Label Grid.Row="2" Grid.Column="0" Content="PI Point path" />
        <Label Grid.Row="3" Grid.Column="0" Content="Value to write" />
        <TextBox Grid.Row="0" Grid.Column="1" Margin="3"  Name="userNameTextBox" />
        <PasswordBox Grid.Row="1" Grid.Column="1" Margin="3" Name="pwBox" />
        <TextBox Grid.Row="2" Grid.Column="1" Margin="3"  Name="tagTextBox" />
        <TextBox Grid.Row="3" Grid.Column="1" Margin="3"  Name="valueTextBox" />
        <Button Grid.Row="4" Grid.Column="1" Margin="5" Content="Write" Name="writeBtn" Click="writeBtn_Click"/>
        <TextBlock Grid.Row="5" Grid.Column="1" Margin="3" Name="statusTextBlock" TextWrapping="WrapWithOverflow" />
    </Grid>
</Window>

 

The window accepts username and password combination and use basic authentication to connect to PI Web API. Any error (or success) messages are displayed in the bottom of window. It will first try to use the tag path specified to get to the endpoint that accepts a value-writing POST request. In this simple example, we will only be supplying the value in the payload. This means that the value will be written in current time.

namespace piwebapi_cs_wpf_test
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        public MainWindow()
        {
            InitializeComponent();
        }

        /* This method takes the username and password specified to use basic authentication to connect to 
         * PI Web API. It then attempts to resolve the tag path provided and write to the tag. */
        private async void writeBtn_Click(object sender, RoutedEventArgs e)
        {
            string baseUrl = "https://dng-code.osisoft.int/piwebapi";
            string userName = userNameTextBox.Text;
            string password = pwBox.Password;
            string tagPath = tagTextBox.Text;
            PIWebAPIClient piWebAPIClient = new PIWebAPIClient(userName, password);

            try
            {
                //Resolve tag path
                string requestUrl = baseUrl + "/points/?path=" + tagPath;
                Task<JObject> tget = piWebAPIClient.GetAsync(requestUrl);
                statusTextBlock.Text = "Processing...";

                //Attempt to write value to the tag
                Object payload = new
                {
                    value = valueTextBox.Text
                };
                string data = JsonConvert.SerializeObject(payload);
                JObject jobj = await tget;
                await piWebAPIClient.PostAsync(jobj["Links"]["Value"].ToString(), data);

                //Display final results if successful
                statusTextBlock.Text = "Write success!";
            }
            catch (HttpRequestException ex)
            {
                statusTextBlock.Text = ex.Message;
            }
            catch (Exception ex)
            {
                statusTextBlock.Text = ex.Message;
            }
            finally
            {
                //We are closing the HttpClient after every write in this simple example. This is not necessary.
                piWebAPIClient.Dispose();
            }
        }
    }
}

 

Let's try to write to a tag!

 

Conclusion

The full VS 2013 solution (including the test project) is in the GitHub repository piwebapi-cs-helper. HttpClient looks pretty easy to use so far, and it works well with PI Web API. I am curious to see what others in the community have been using to access a PI Web API (or other REST API) in C#. Comments and feedback are welcome!

Introduction

 

Wikipedia as you know is a free online encyclopedia. Depending on which type of application you are developing, integrating with it may have a lot of benefits. Last year, we have developed a web site that shows the weather information from different cities of the world. The time-series data was stored of course on the PI System. A cool feature we have added was to show the city description from wikipedia. I am sure you can find out other use cases. The question I will try to answer is how to integrate it with the PI System.

 

On this first part I will show you how to develop a .NET library to get data from Wikipedia. On the second part, we are going to develop an AF custom data reference for this use case.

 

The source code package of the first part can be download here.

 

MediaWiki Web API

 

As Wikipedia is actually a Media Wiki installation, it provides direct, high-level access to the data contained in its databases. Clients can log into a wiki, get data, post changes automatically by making HTTP requests.

 

The API provides a large number of options. This is what we want to do:

 

  1. Get the title of the wiki page found.
  2. Get the first paragraph of the wiki page.
  3. Get the thumbnail of the wiki page.
  4. Get the coordinates of the wiki page. This will only work if the wiki page is about something physical which has a fixed position like a tourist attraction.

 

Here in Sao Paulo (Brazil), there is a famous park called Ibirapuera Park. Let's use it on the following examples:

 

You can get the title using the URL: https://en.wikipedia.org/w/api.php?action=query&format=json&titles=Ibirapuera%20Park. This will return a JSON response:

 

 {"batchcomplete":"","query":{"pages":{"3368541":{"pageid":3368541,"ns":0,"title":"Ibirapuera Park"}}}}

 

This means that it has found a wiki page whose pageid is 3368 and whose title is "Ibirapuera Park". Getting the title is the first step to get the other information: first paragraph, thumbnail and coordinates.

 

Using the title, we have the URL address of the wiki page: http://en.wikipedia.org/wiki/ + "encoded title". This results on "http://en.wikipedia.org/wiki/Ibirapuera%20Park".Using HtmlAgilityPack you can extract the first paragraph. I will show you how to do that on the next section.

 

In order to get the thumbnail of the wiki page, you should access the following URL: https://en.wikipedia.org/w/api.php?action=query&titles=Ibirapuera%20Park&prop=pageimages&format=json&pithumbsize=100. You can specify the format of the response and the size of the thumbnail by setting up the proper query strings. The JSON response would be:

 

{"batchcomplete":"","query":{"pages":{"3368541":{"pageid":3368541,"ns":0,"title":"Ibirapuera Park","thumbnail":
{"source":"https://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Oscar_Niemeyer%2COca%2C_Bienal%2C_Audit%C3%B3rio.jpg/100px-Oscar_Niemeyer%2COca%2C_Bienal%2C_Audit%C3%B3rio.jpg","width":100,"height":50},"pageimage":"Oscar_Niemeyer,Oca,_Bienal,_Audit\u00f3rio.jpg"}}}}

 

If you access the source attribute, it will show the image below:

 

100px-Oscar_Niemeyer%2COca%2C_Bienal%2C_Audit%C3%B3rio.jpg

 

 

Finally, in order to have access to the geolocation of the Ibirapuera Park, you can access the following URL:  http://en.wikipedia.org/w/api.php?action=query&prop=coordinates&format=json&colimit=10&exportnowrap=&iwurl=&titles=Ibirapuera%20Park.

 

This will result on the following JSON response:

 

{"batchcomplete":"","query":{"pages":{"3368541":{"pageid":3368541,"ns":0,"title":"Ibirapuera Park","coordinates":[{"lat":-23.5883,"lon":-46.6589,"primary":"","globe":"earth"}]}}}}

 

We can conclude that according to Wikipedia, the Ibirapuera Park in São Paulo is located with a latitude = -23.5883 and longitude = -46.6589.

Developing our library

 

Now that you have an idea about how to query against Wikipedia API, let's create a new project which is Class Library on Visual Studio. Rename the default .cs file to WikipediaWebService.cs.

 

Add two Nuget packages using the following commands:

 

  • Install-package HtmlAgilityPack
  • Install-package Newtonsoft.Json

 

We have some methods that query wikipedia. Most of them gets the title first and then makes another request to get the desired information. Two methods are available for getting data of the web: GetStringResponse() and GetJsonResponse(). Both have the url as an input. GetJsonResponse() calls GetStringResponse() and then converts the result to a dynamic object through some methods provided by the Newtonsoft.Json library.

 

Concerning getting the description of the first paragraph, I am taking a different approach than accessing the API. We transfer the HTML string using the GetStringResponse() and then loads it on the Html Agility Pack library. This library, which enables you to find selected nodes, was used on one of my blog posts Creating an AF Elements Tree using web page content. In our case, the first paragraph is a div nose whose class name is "mw-content-ltr".I have taken the StripHTML() static method from the web in order to convert the HTML node to a text. And this is how we get the description of the wiki page. You can find similar results by accessing the API directly but I wanted to show you a different approach.

 

As this is just an example, I am not handling the exceptions properly. If I receive an exception it will return a null or an empty object depending on each situation.

 

Two additional class for coordinates and image properties shall be created:

 

    public class Coordinate
    {
        public double Latitude { get; set; }
        public double Longitude { get; set; }
    }

    public class Image
    {
        public string Title { get; set; }
        public string Description { get; set; }
        public string ThumbnailUrl { get; set; }
        public string ImageUrl{ get; set; }
        public string UserName { get; set; }
    }

 

The code snippet forthe WikipediaWebService class is below.

 

 

using HtmlAgilityPack;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using System.Web;




namespace WikipediaDR
{
    public class WikipediaWebService
    {


        private static string GetWikipediaTitle(string query, out string wikiPageId)
        {
            wikiPageId = string.Empty;
            string title = string.Empty;
            try
            {
                string url = "http://en.wikipedia.org/w/api.php?action=query&format=json&titles=" + HttpUtility.UrlPathEncode(query.Trim());
                dynamic webResponse = GetJsonResponse(url);
                JObject wikiPage = webResponse["query"]["pages"];
                wikiPageId = wikiPage.Properties().First().Name;
                title = webResponse["query"]["pages"][wikiPageId]["title"];
            }
            catch (Exception)
            {
                return string.Empty;
            }
            return title;
        }


        public static string GetWikipediaDescription(string query, int numberOfParagraphs = 1)
        {
            string wikipediaDescription = string.Empty;
            string wikiPageId = string.Empty;
            try
            {
                string title = GetWikipediaTitle(query, out wikiPageId);
                string url = "http://en.wikipedia.org/wiki/" + HttpUtility.UrlPathEncode(title);
                string webStringResponse = GetStringResponse(url);
                var doc = new HtmlAgilityPack.HtmlDocument();
                doc.LoadHtml(webStringResponse);


                HtmlNode root = doc.DocumentNode;
                HtmlNodeCollection TableNodeCollection = root.SelectNodes("//div[contains(@class,'mw-content-ltr')]").First().SelectNodes("p"); ;


                wikipediaDescription = StripHTML(TableNodeCollection[0].OuterHtml.ToString());
                for (int i = 1; i < numberOfParagraphs - 1; i++)
                {
                    wikipediaDescription += "\n" + StripHTML(TableNodeCollection[i].OuterHtml.ToString());
                }
            }
            catch (Exception)
            {
                return string.Empty;
            }
            return wikipediaDescription;
        }






        public static string GetWikipageThumbnail(string query, int widthPixel)
        {
            string wikiPageId = string.Empty;
            string wikipageThumbnailUrl = string.Empty;
            try
            {
                string title = GetWikipediaTitle(query, out wikiPageId);
                string url = "http://en.wikipedia.org/w/api.php?action=query&titles=" + HttpUtility.UrlPathEncode(title) + "&prop=pageimages&format=json&pithumbsize=" + widthPixel.ToString();
                dynamic webResponse = GetJsonResponse(url);
                wikipageThumbnailUrl = webResponse["query"]["pages"][wikiPageId]["thumbnail"]["source"].ToString();
            }
            catch (Exception)
            {
                return string.Empty;
            }
            return wikipageThumbnailUrl;
        }




        public static Image GetWikipediaImageProperties(string imageTitle)
        {
            Image image = new Image();
            image.Title = imageTitle;
            string wikiPageId = string.Empty;
            string wikipageThumbnailUrl = string.Empty;
            try
            {
                string url = "http://en.wikipedia.org//w/api.php?action=query&prop=imageinfo&format=json&iiprop=user%7Curl&iilimit=10&iiurlwidth=100&iiurlheight=100&exportnowrap=&iwurl=&titles=" + HttpUtility.UrlPathEncode(imageTitle);
                dynamic webResponse = GetJsonResponse(url);
                image.UserName = webResponse["query"]["pages"]["-1"]["imageinfo"][0]["user"].ToString().ToLower();
                image.ImageUrl = webResponse["query"]["pages"]["-1"]["imageinfo"][0]["url"].ToString().ToLower();
                image.Description = webResponse["query"]["pages"]["-1"]["imageinfo"][0]["descriptionurl"].ToString().ToLower();
                image.ThumbnailUrl = webResponse["query"]["pages"]["-1"]["imageinfo"][0]["thumburl"].ToString().ToLower();
            }
            catch (Exception)
            {
                return null;
            }


            return image;
        }




        public static Coordinate GetWikipediaCoordinates(string query)
        {
            Coordinate coordinate = new Coordinate();
            string wikiPageId = string.Empty;
            string wikipageThumbnailUrl = string.Empty;


            try
            {
                string title = GetWikipediaTitle(query, out wikiPageId);
                string url = "http://en.wikipedia.org/w/api.php?action=query&prop=coordinates&format=json&colimit=10&exportnowrap=&iwurl=&titles=" + HttpUtility.UrlPathEncode(title);
                dynamic webResponse = GetJsonResponse(url);
                string lat = webResponse["query"]["pages"][wikiPageId]["coordinates"][0]["lat"].ToString();
                string lon = webResponse["query"]["pages"][wikiPageId]["coordinates"][0]["lon"].ToString();


                coordinate.Latitude = Convert.ToDouble(lat);
                coordinate.Longitude = Convert.ToDouble(lon);


            }
            catch (Exception)
            {
                return null;
            }
            return coordinate;
        }




        public static List<string> GetWikipageImagesTitles(string query)
        {
            List<string> imagesTitles = new List<string>();
            string wikiPageId = string.Empty;
            string wikipageThumbnailUrl = string.Empty;
            try
            {
                string title = GetWikipediaTitle(query, out wikiPageId);
                string url = "http://en.wikipedia.org/w/api.php?action=query&titles=" + HttpUtility.UrlPathEncode(title) + "&prop=images&imlimit=20&format=json";
                dynamic webResponse = GetJsonResponse(url);
                foreach (var image in webResponse["query"]["pages"][wikiPageId]["images"])
                {
                    imagesTitles.Add(image.title.Value.ToString());
                }
            }
            catch (Exception)
            {
   
            }
            return imagesTitles;
        }








        private static dynamic GetJsonResponse(string url)
        {
            return JObject.Parse(GetStringResponse(url));
        }


        private static string GetStringResponse(string url)
        {
            WebRequest request = WebRequest.Create(url);
            WebResponse response = request.GetResponse();
            StreamReader sw = new StreamReader(response.GetResponseStream());
            return sw.ReadToEnd().Replace("/**/", string.Empty);
        }




        private static string StripHTML(string source)
        {
            try
            {
                string result;


                // Remove HTML Development formatting
                // Replace line breaks with space
                // because browsers inserts space
                result = source.Replace("\r", " ");
                // Replace line breaks with space
                // because browsers inserts space
                result = result.Replace("\n", " ");
                // Remove step-formatting
                result = result.Replace("\t", string.Empty);
                // Remove repeating spaces because browsers ignore them
                result = System.Text.RegularExpressions.Regex.Replace(result,
                                                                      @"( )+", " ");


                // Remove the header (prepare first by clearing attributes)
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<( )*head([^>])*>", "<head>",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"(<( )*(/)( )*head( )*>)", "</head>",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         "(<head>).*(</head>)", string.Empty,
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                // remove all scripts (prepare first by clearing attributes)
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<( )*script([^>])*>", "<script>",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"(<( )*(/)( )*script( )*>)", "</script>",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                //result = System.Text.RegularExpressions.Regex.Replace(result,
                //         @"(<script>)([^(<script>\.</script>)])*(</script>)",
                //         string.Empty,
                //         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"(<script>).*(</script>)", string.Empty,
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                // remove all styles (prepare first by clearing attributes)
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<( )*style([^>])*>", "<style>",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"(<( )*(/)( )*style( )*>)", "</style>",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         "(<style>).*(</style>)", string.Empty,
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                // insert tabs in spaces of <td> tags
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<( )*td([^>])*>", "\t",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                // insert line breaks in places of <BR> and <LI> tags
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<( )*br( )*>", "\r",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<( )*li( )*>", "\r",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                // insert line paragraphs (double line breaks) in place
                // if <P>, <DIV> and <TR> tags
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<( )*div([^>])*>", "\r\r",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<( )*tr([^>])*>", "\r\r",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<( )*p([^>])*>", "\r\r",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                // Remove remaining tags like <a>, links, images,
                // comments etc - anything that's enclosed inside < >
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"<[^>]*>", string.Empty,
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                // replace special characters:
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @" ", " ",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&bull;", " * ",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&lsaquo;", "<",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&rsaquo;", ">",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&trade;", "(tm)",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&frasl;", "/",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&lt;", "<",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&gt;", ">",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&copy;", "(c)",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&reg;", "(r)",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                // Remove all others. More can be added, see
                // http://hotwired.lycos.com/webmonkey/reference/special_characters/
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         @"&(.{2,6});", string.Empty,
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                // for testing
                //System.Text.RegularExpressions.Regex.Replace(result,
                //       this.txtRegex.Text,string.Empty,
                //       System.Text.RegularExpressions.RegexOptions.IgnoreCase);


                // make line breaking consistent
                result = result.Replace("\n", "\r");


                // Remove extra line breaks and tabs:
                // replace over 2 breaks with 2 and over 4 tabs with 4.
                // Prepare first to remove any whitespaces in between
                // the escaped characters and remove redundant tabs in between line breaks
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         "(\r)( )+(\r)", "\r\r",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         "(\t)( )+(\t)", "\t\t",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         "(\t)( )+(\r)", "\t\r",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         "(\r)( )+(\t)", "\r\t",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                // Remove redundant tabs
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         "(\r)(\t)+(\r)", "\r\r",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                // Remove multiple tabs following a line break with just one tab
                result = System.Text.RegularExpressions.Regex.Replace(result,
                         "(\r)(\t)+", "\r\t",
                         System.Text.RegularExpressions.RegexOptions.IgnoreCase);
                // Initial replacement target string for line breaks
                string breaks = "\r\r\r";
                // Initial replacement target string for tabs
                string tabs = "\t\t\t\t\t";
                for (int index = 0; index < result.Length; index++)
                {
                    result = result.Replace(breaks, "\r\r");
                    result = result.Replace(tabs, "\t\t\t\t");
                    breaks = breaks + "\r";
                    tabs = tabs + "\t";
                }


                // That's it.
                return result;
            }
            catch
            {
                return string.Empty;
            }
        }
    }
}

Unit Testing

 

We could test the WikipediaWebService class by create a new Console Application project. I will take a different approach and use Unit Testing instead whose libraries Visual Studio already provides.

 

According to Microsoft, the primary goal of unit testing is to take the smallest piece of testable software in the application, isolate it from the remainder of the code, and determine whether it behaves exactly as you expect by defining TestMethods.

 

In order to create a new unit test project, right click on the solution --> Add --> Project.. You will find the "Unit Test Project" under the Test section within Visual C#.  Once the project whose name is WikipediaDR.Tests is created, you can define its method on the WikipediaWebServiceUnitTest.

 

 

 

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Collections.Generic;
using System.Threading.Tasks;


namespace WikipediaDR.Tests
{
    [TestClass]
    public class WikipediaWebServiceUnitTest
    {
        private string GetTestQuery()
        {
            return "Ibirapuera Park";
        }


        [TestMethod]
        public void WikipediaDescriptionTest()
        {
            string query = GetTestQuery();
            string wikipediaDescription = WikipediaWebService.GetWikipediaDescription(query, 1);
            Assert.IsTrue(wikipediaDescription.Length > 100);
        }


        [TestMethod]
        public void WikipediaThumbnailTest()
        {
            string query = GetTestQuery();
            string wikipageThumbnail = WikipediaWebService.GetWikipageThumbnail(query, 100);
            Assert.IsTrue(wikipageThumbnail.Contains(".org"));
            Assert.IsTrue(wikipageThumbnail.Contains("http"));
            Assert.IsTrue(wikipageThumbnail.Contains(".png") || wikipageThumbnail.Contains(".jpg"));
        }


        [TestMethod]
        public void WikipediaImagesTest()
        {
            string query = GetTestQuery();
            List<string> imagesTitleList = WikipediaWebService.GetWikipageImagesTitles(query);
            foreach (string image in imagesTitleList)
            {
                Assert.IsTrue(image.Contains("File"));
                Assert.IsTrue(image.ToLower().Contains(".png") || image.ToLower().Contains(".svg") || image.ToLower().Contains(".jpg"));
            }
        }
        [TestMethod]
        public void WikipediaCoordinateTest()
        {
            string query = GetTestQuery();
            Coordinate coordinate = WikipediaWebService.GetWikipediaCoordinates(query);
            Assert.IsNotNull(coordinate);
            Assert.IsNotNull(coordinate.Latitude);
            Assert.IsNotNull(coordinate.Longitude);
            Assert.IsTrue(coordinate.Latitude <= 90);
            Assert.IsTrue(coordinate.Latitude >= -90);
            Assert.IsTrue(coordinate.Longitude <= 180);
            Assert.IsTrue(coordinate.Longitude >= -180);


        }
        [TestMethod]
        public void WikipediaImagesPropertiesTest()
        {
            string query = GetTestQuery();
            List<string> imagesTitleList = WikipediaWebService.GetWikipageImagesTitles(query);
            Parallel.ForEach(imagesTitleList, imageTitle =>
            {
                Image image = WikipediaWebService.GetWikipediaImageProperties(imageTitle);
                if (image != null)
                {
                    Assert.IsNotNull(image);
                    Assert.IsTrue(image.Title == imageTitle);
                    Assert.IsTrue(image.Description.Length > 0);
                    Assert.IsTrue(image.UserName.Length > 0);


                    Assert.IsTrue(image.ImageUrl.Contains(".org"));
                    Assert.IsTrue(image.ImageUrl.Contains("http"));
                    Assert.IsTrue(image.ImageUrl.Contains(".png") || image.ImageUrl.Contains(".jpg") || image.ImageUrl.Contains(".svg"));


                    Assert.IsTrue(image.ThumbnailUrl.Contains(".org"));
                    Assert.IsTrue(image.ThumbnailUrl.Contains("http"));
                    Assert.IsTrue(image.ThumbnailUrl.Contains(".png") || image.ThumbnailUrl.Contains(".jpg") || image.ThumbnailUrl.Contains(".svg"));
                }
            });


        }
    }
}

 

Five unit tests were defined. When you run them, they will pass only if they don't throw any exception. Otherwise, they will fail. The Assert class provides many static methods to check if the objects behave the way you expect. Let's take an example:

 

Assert.IsTrue(image.ImageUrl.Contains(".org"));

 

if the image.ImageUrl property which is a string does not contain ".org" within, then it will throw an exception and the method will fail.

 

Clicking on "Run All" button on Test Explorer pane will run all the tests with a single click. Make sure they all pass their tests!

 

image.jpg

 

Conclusions

 

Our WikipediaWebService class is ready. If you have an input string, it will search for a wikipage and returns its title in case a wikipage is found. Other methods were added in order to get the thumbnail and image properties from the wikipage found. Finally, we have created a unit test class for testing our library.

 

The next part we will develop a custom AF data reference which will call the WikipediaWebService class in order to obtain the Wikipedia description. Stay tuned!

Filter Blog

By date: By tag: