Last week I delivered a SAS Talks session: Introduction to Custom Tasks for SAS Enterprise Guide (click to watch it). I promised that I would share a collection of resources where you can learn more information.

First, here is a round-up of the custom tasks that I mentioned, with links for more information and downloading. These are tasks that you can use today in SAS Enterprise Guide.

Now, a few selections from the Q&A section of the talk.

Does my SAS license allow me to run custom tasks?
Yes, there are no license restrictions for running custom tasks (as long as you have SAS Enterprise Guide and/or the SAS Add-In for Microsoft Office). However, the work that the task performs might leverage a SAS capability that you don't have. For example, if you found a custom task that created a forecast using PROC ESM, you would need SAS/ETS in order to benefit from that task.

Note: the SAS OnDemand for Professionals and SAS OnDemand for Academics offerings use special versions of SAS Enterprise Guide in a SAS learning environment. Custom tasks are not usable in that environment.

Do I have to register a custom task in SAS Management Console before I can use it (or give it to my team to use)?
You need to register a custom task in SAS Management Console only if you want to limit who has access to it. By default, any SAS Enterprise Guide user can install and run a custom task simply by "installing" the custom task DLL(s) to one of several designated local folders.

I'm running an older version of SAS Enterprise Guide. Can I still use the custom tasks that you described during your talk?
The answer is: it depends. Some of the tasks I showed (such as the Top N Report and the Soduko Solver) are designed to work with SAS Enterprise Guide 4.1, and they work in all later versions of SAS Enterprise Guide as well. Other tasks (such as Project Reviewer) were built for SAS Enterprise Guide 4.3, and still work with version 5.1. See the documentation that accompanies the task for which versions of the software are supported.

Can you describe the difference between SAS stored processes and custom tasks?
A SAS stored process is a SAS program that is stored in a central location, registered in the SAS Metadata environment. You can run a stored process in SAS Enterprise Guide, SAS Add-In for Microsoft Office, in web apps like SAS Web Report Studio and even SAS Visual Analytics. But as a stored process author, you have limited control over the prompts and user interface experience. A custom task runs only in SAS Enterprise Guide and the SAS Add-In for Microsoft Office. Custom tasks can run SAS programs (like a stored process) but they can be used for many other operations that aren't necessarily captured in a SAS program. And, as a custom task author, you have complete control over the user interface.

For a summary of the differences between custom tasks, stored processes, and other methods of packaging custom processes, see Chapter 1: Why Custom Tasks. It's the free chapter from Custom Tasks for SAS Enterprise Guide Using Microsoft .NET.

What's the best way to get started with building my own custom task?
It's definitely easier to use a custom task than it is to build one. But building tasks can be a fun and rewarding experience too. To get started, my shameless advice is to begin with my book: Custom Tasks for SAS Enterprise Guide Using Microsoft .NET. The information in the book will save you loads of time in research, and should provide you with just the right amount of information to get started.

Of course, you can also visit go.sas.com/customtasksapi for the "raw" information about the custom task APIs and lots of example projects.

If you missed the talk and you still want to learn about how custom tasks can help you to do more with SAS, go watch the recording right now!

Revenge of the Nerds was so 1980s. Now it's a new world order: math geeks and athletes are working together.

I'm not talking just about what happens when data nerds observe, analyze, and predict sports outcomes -- as they do in March Madness with their "bracketology". That's compelling, but your ability to predict an outcome is nothing when compared with the ability to influence the outcome.

That's what my friend Drew Cannon is doing for the Butler Bulldogs, and even Sports Illustrated has noticed the impact.

I first mentioned Drew in a blog 3 years ago, when his work as a summer intern for a basketball scouting publication earned him recognition in a New York Times article.

Today, he's on the bench as an assistant coach, diligently recording every aspect of the game that can be counted and measured. He then compiles the data, analyzes it, and uses it to inform the head coach via a system of "rules": for example, which players work well together in which situations. When the rules are followed, there's a better outcome. From the Sports Illustrated article:

"I love it when Drew [reminds] a guy who coached in two national championship games [of these rules]," Butler assistant Michael Lewis said. "He'll say, 'When we played by the rules, we were plus 5. When we didn't we were minus 3.' I love it, for a 22-year old kid to have the guts to say it."

Guts? Yes, I believe it. But when you've got the data and a solid analysis, it's a lot easier to muster your nerve.

Are you a new SAS user who isn't sure where to begin? Have no fear, because you're not alone. Here at SAS, we often receive questions from people who need help getting started with the software.

Getting Started with SAS is the topic of a SAS Talks session that I will co-host with SAS instructor Stacey Syphus on April 11, 2013 at 1pm ET. Tune in if you can, but don't worry if you can't -- the session will be recorded and available later as part of the SAS Talks series.

The goals of this session are to familiarize you with the content of typical SAS programs, teach you to read SAS log output, and show you how to run and modify programs. You won't become a SAS programming expert by attending this session, but it should provide a good foundation for further study. If you want to find out more, dozens of SAS programming books and training courses are available that cover every aspect of the craft. (And yes, SAS programmers often do regard themselves as craftspeople, of a sort.)

In this session, Stacey and I will cover:

• a high-level review of SAS, its foundational technologies (including Base SAS) and how it works.
• the different methods for working with SAS, including the SAS windowing environment and SAS Enterprise Guide.
• how to determine which SAS approach is best for you.

Also, you'll learn where you can find resources at SAS that are available specifically for SAS beginners (including my SAS for Dummies book, now available in its second edition).

I hope that you'll join Stacey and me for the session. Don't wait to begin getting the most out of SAS.

Related links:

Getting started with SAS Enterprise Guide
SAS Enterprise Guide for the Programmer: What's in It for Me?

In a previous article, I described how to use Windows PowerShell to connect to a remote SAS Workspace, run a SAS program, and retrieve the results. In that example, the only results I retrieved were the SAS log and SAS listing (text) output, if any.

When you run a SAS program on a remote session, sometimes the expected result is more substantial than just a text-based listing. What if your program creates Output Delivery System (ODS) files such as HTML or graphics? To retrieve those results, you need to download the files from the SAS session to your local machine, where your application can access them. Consider this program, implemented in a PowerShell script, which creates an image with PROC SGPLOT:

# change these to your own SAS-session-based
# file path and file name
# Note that $destImg can't be > 7 chars
$destPath = "/projects/results"
$destImg = "hist"

# program to run
# could be read from external file
$program =  
       "ods graphics / imagename='$destImg';
        ods listing gpath='$destPath' style=plateau;
        proc sgplot data=sashelp.cars;
        histogram msrp;
        density msrp;
        run;"

# run the program
$objSAS.LanguageService.Submit($program);

Downloading a file with FileService

The SAS Workspace provides the FileService API, which allows you to transfer file-based content between your local application and the SAS session. As with most file-based operations in SAS, the FileService relies on the use of a SAS fileref, or the name that SAS uses to reference your file within a program. For a file download operation, these are the basic steps:

• Obtain a SAS reference to the file (FileService AssignFileref method.
• Tell SAS to open the remote file for reading (OpenBinaryStream method).
• Read the contents of the file into a local array of bytes (Read method, repeating in 1K increments).
• Write the contents into a local file (PowerShell objects).
• When completed, close the local and remote file handles, and unassign the SAS file reference (Close method and DeassignFileref method).

Here's a PowerShell program snippet that implements these steps. (The complete example is available on GitHub.)

# local directory for downloaded file
$localPath = "c:\temp"

# now download the image file
$fileref = ""

# assign a Fileref so we can use FileService from IOM
$objFile = $objSAS.FileService.AssignFileref(
     "img", "DISK", "$destPath/$destImg.png", 
     "", [ref] $fileref);

$StreamOpenModeForReading = 1
$objStream = $objFile.OpenBinaryStream($StreamOpenModeForReading)

# define an array of bytes
[Byte[]] $bytes = 0x0

$endOfFile = $false
$byteCount = 0
$outStream = [System.IO.StreamWriter] "$localPath\$destImg.png"
do
{
  # read bytes from source file, 1K at a time
  $objStream.Read(1024, [ref]$bytes)
  
  # write bytes to destination file
  $outStream.Write($bytes)
  # if less than requested bytes, we're at EOF
  $endOfFile = $bytes.Length -lt 1024
  
  # add to byte count for tally
  $byteCount = $byteCount + $bytes.Length
  
} while (-not $endOfFile)

# close input and output files
$objStream.Close()
$outStream.Close()

# free the SAS fileref
$objSAS.FileService.DeassignFileref($objFile.FilerefName)

Write-Output "Downloaded $localPath\$destImg.png: SIZE = $byteCount bytes"

NOTE: PROCEDURE SGPLOT used (Total process time):
      real time           13.18 seconds
      cpu time            0.12 seconds
      
NOTE: Listing image output written to /projects/result/hist.png.
NOTE: There were 428 observations read from the data set SASHELP.CARS.

Downloaded c:\temp\hist.png: SIZE = 15092 bytes

Related links

• Complete code example on GitHub (SasWorkspaceDownloadFile.ps1)
• Running Windows PowerShell scripts
• Using Windows PowerShell to connect to a SAS Workspace server
• Copying files in SAS Enterprise Guide
• Create Your Own Client Apps using SAS Integration Technologies

Getting Started with SAS -- a free webinar for new SAS users -- is now available for on-demand viewing.

During this SAS Talks session, Stacey Syphus and I review the basics of what SAS is, what people use it for, and what you can expect to see as a new user. We talk about SAS programs: what they look like and what they can be used for. You won't learn how to write programs by watching this short webinar, but you'll at least know a SAS program when you see it.

I also demonstrate two of the most common SAS interfaces:

• The SAS windowing environment (a.k.a. "display manager", and a.k.a "base SAS")
• SAS Enterprise Guide

(It's been a long time since I showed off the SAS windowing environment. Within this blog I share lots of information about SAS Enterprise Guide, and don't devote much time to the more traditional interface for SAS. But the fact is each of these tools are in wide use around the world, and a "Getting Started" talk would be incomplete if we didn't include both.)

And as long-time SAS users know, learning to use SAS effectively is more than just learning a skill. It also involves getting "plugged in" to the exceptional SAS user community -- whether that's reading discussion forums, attending user group meetings, or working towards SAS certification.

If you are just getting started with SAS, or if you have colleagues who would appreciate a gentle introduction, give this webinar a try. It could be your first step into an exciting career!

In the SAS User Groups LinkedIn group, some generous "old timers" offer tips to the potentially shy newcomers for connecting with other SAS professionals at SAS Global Forum. Perhaps these folks remember their own introverted natures, and they want to encourage attendees to get the most out of their conference experience.

One group participant, David Corliss, offered an inspiring (but not unusual!) account of how he met a fellow SAS user whose specific job has a big impact on David's family. I'm sharing the story here (with David's permission).

So, I just have to tell this story about meeting a first-timer at [SAS Global Forum] 2012. I was standing in line to buy a bagel at the hotel convenience store. The person in front of me saw my badge with the tags for speaker and session coordinator - he figured I had been around a little and wondered if I would look at his schedule and make any recommendations.

Naturally, I asked what his background was and what he wanted to accomplish. He said he was a statistician with the CDC in Altanta, working on infectious diseases in the third world. Due to cost restrictions, it is common in such places to receive only partial treatments for diseases requiring lengthy treatment, such as TB. His team calculated the effectiveness of these partial treatments and made recommendations to complete the treatment should it become available later on. He wanted my advice on papers and classes to take in so he could do more to save uncounted thousands of lives....

...like that of my own daughter, adopted from Russia at age 7 and exposed to TB as a small child. I owed this first-timer as inestimable debt of gratitude as the person who determined the remedial treatment needed to keep her as safe as possible from developing TB later on.

At [SAS Global Forum], whether you are a first timer or have been there for years and years, be sure to meet as many people as you can. You never know who you are going to meet.

It's true -- you never know. I'm inspired by SAS users at every conference I attend. I hope to see you there -- and to be inspired again!

Even if you cannot attend SAS Global Forum next week, you can experience it virtually via the Livestream sessions.

This year I will reprise my role as host of SAS Tech Talks, a pair of live webcasts that feature SAS R&D professionals and their latest technological wares. Click "play" on the viewer below to see what's happening right now. Below this widget, you can see the scheduled SAS Tech Talks lineup. (Note that scheduled times are in the Pacific USA timezone -- coming to you from San Francisco!)

Monday, April 29 at 12:30 p.m. PT

SAS Web Editor
Mike Monaco, Director, Web SAS Technologies R&D

SAS Visual Analytics Explorer
Falko Schulz, Principal Software Developer, SAS BI Visualization R&D

SAS Mobile BI
Himesh Patel, Senior Director, SAS Data Visualization R&D

SAS program language advancements
Rick Langston, Senior Manager, SAS Platform R&D

DataFlux and Data Integration
Nancy Rausch, Senior Manager, SAS Data Management R&D

Tuesday, April 30 at 10:30 a.m. PT

SAS Marketing Automation
Brian Chick, Senior Manager, Customer Intelligence R&D

SAS App Works
Marty Tomasi, Director, SAS Middle Tier Platform R&D

ODS Graphics
Sanjay Matange, Director, SAS Scientific Visualization R&D

SAS Visual Statistics
Tonya Balan, Director, SAS Analytics Product Management

I've known most of these folks for a long time, and according to my quick math they possess well over 150 years of SAS experience (cumulatively, that is -- not each). And they all still work on cutting-edge projects that you'll want to learn more about!

SAS Global Forum is like a BIG buffet of SAS knowledge, all served up in just a few days. There are so many good presentations and activities, but you cannot fit them all on your plate. You have to choose.

To make the choices easier, the SAS Global Forum committee has assembled a collection of presentations that you can take "to go". It's called SAS Global Forum Take-Out. (In some countries, you might prefer the term "take-away" -- but you get the idea.)

Hosted at Brainshark.com, these presentations are "performed" by the actual authors of select SAS Global Forum papers. You can view the slides and notes, and listen to the expert present the material -- whenever and wherever you want. And these are available to everyone -- even if you can't attend the conference.

I was fortunate enough to have a presentation selected to be part of Take-Out. It's called For All of the Hats You Wear: SAS Enterprise Guide Has Got You Covered.

I hope that I have a full and lively audience for my presentation at the conference. But I'm happy that the work I've assembled can "live on" and reach a wider set of people around the world. With Take-Out, you can always reach into the back of the SAS Global Forum "fridge" and find something fresh for your brain to munch on.

Links

SAS Global Forum Take-Out presentations on Brainshark.com

What a conference so far...

SAS Global Forum is now in full swing, and the two big sessions (Opening Session last night, and the Tech Session this morning) have been packed with impressive material. The attendees I've spoken with have been suitably impressed and enthused. You can view the sessions all on the Livestream channels.

I'll post more when I have a chance, but for now you can follow along on the SAS Users Groups blog, the Twitter stream, and on the Livestream.

Now it's almost time for me to hit the makeshift studio and broadcast the first of two SAS Tech Talks. Tune in if you can!

Even though it's been around for well over a decade, SAS Enterprise Guide was still a hot topic among attendees at SAS Global Forum this year.

In the Technology Connection -- the big session on Monday morning -- SAS R&D staff used the conference agenda content to demonstrate the power of SAS Text Miner. By categorizing the papers for this year and comparing to previous years, you can see the continued growing interest in several key topics, including SAS Enterprise Guide.

I captured this (grainy) screenshot from the Livestream archive of the Technology Connection. Enterprise Guide papers are represented by the yellow bar in this screen shot from SAS Visual Analytics, showing 4 years of conference data:

I presented one of those papers: For All the Hats You Wear: SAS Enterprise Guide Has Got You Covered. In the presentation (which you can watch on SAS Global Forum Take-out), I describe several types of users who accomplish work in SAS Enterprise Guide, including:

• The Newbie
• The Business Analyst
• The Programmer
• The Statistician
• The Data Scientist
• The Administrator
• The Consultant

I've done my share of blogging and presenting about SAS Enterprise Guide over the years, but with over 80 papers or posters that addressed it at this year's conference, it's obvious that others are also keen to share their experiences. That's great, because I have an obvious bias when I describe SAS Enterprise Guide as an essential tool for SAS users; you no longer have to simply take my word for it.

I've also seen others sharing on their own blogs. For example, here's a series from the bi-notes.com blog:

• These are a Few of My Favorite Things… about the SAS Enterprise Guide Program Editor
• PC SAS Programmer Facing UNIX? SAS Enterprise Guide to the Rescue
• SAS Enterprise Guide Process Flows: The Good, The Bad, and The Ugly
• SAS Enterprise Guide: Are You a Member of Programmers Anonymous?

And another series from OptimalBI in New Zealand:

• Enterprise guide for SAS coders. (Part 1 – Code glorious code)
• Enterprise guide for SAS coders.(Part 2 – Exempli Gratia)
• Enterprise guide for SAS coders.(Part 3a – Hotel Paradiso)
• Enterprise guide for SAS coders.(Part 3b – Hotel Paradiso cont.)

I'm happy to see that the SAS user community now creates, sustains, and propagates some really excellent information on these topics. Keep up the great work!

A few months ago I released the Copy Files task for use with SAS Enterprise Guide. The task allows you to transfer any files between your PC and a SAS Workspace session, much like an FTP process. It doesn't rely on FTP though; it uses a combination of SAS code, Windows APIs, and SAS Integration Technologies to get the job done.

It's proven to be a very popular task, because it can be useful in so many situations. It even earned a mention in a SAS Global Forum paper this year (and no, it wasn't a paper that I wrote).

Today I'm going to point out the things that the task doesn't do so well. Or at least, that it didn't do well until I made some updates. My changes were based on two "complaints" from several SAS users.

Read on for the details. But if you don't care and you just want the latest version of the task, you can download it from here.

Complaint #1: Wildcards that are a little too "wild"

The task allows you to use wildcard characters in your file specifications so that you can match multiple files to transfer. A problem occurs though, when your file specification looks like this:

/usr/local/data/*.xls

The SAS macro experts are already shouting out the answer to fix this: use %STR to wrap the slash and "hide" the token from the SAS parser. That's a great idea! Except that the task relies on the SAS "internal" value for this value --and not the displayed value -- when it comes time to process. These values are different when %STR wraps a special character like the forward slash. The macro facility changes out this character with a hexadecimal character called a delta character.

To illustrate, I used another popular custom task -- the SAS Macro Variable Viewer -- to show the inner value of a SAS macro variable:

Notice the funky arrow characters. Is that what you were expecting?

Now the task detects the presence of a forward slash (and some other special characters) and will automatically add the %STR so you don't have to. (But you can still use %STR if you want to.) And it correctly detects the delta characters, if present, to convert them back to their correct form before trying to use the value.

Complaint #2: Fixing line-ending characters but breaking other stuff

Users of FTP might be familiar with binary versus ASCII mode for file transfers. Because UNIX line-endings are different than Windows line-endings for text files, transferring a file in ASCII mode helps to ensure proper line-ending behavior for the target host.

The Copy Files task transfers ALL files using a binary mode. Why? Because in today's global workplace even text-based files often don't adhere to the limited English-centric ASCII standard. Attempting a text-based file transfer could result in encoding mismatches, so it's much safer to transfer content as "binary blobs".

But you still want your text files to have the proper line endings for the target host. To answer that, the Copy Files task offers a "Fix line-ending characters" option that does the following:

• Scans the file to determine whether it's a text file. (This relies on the file content and not on special file extensions such as .TXT or .CSV.).
• Rewrites the file and replaces the line-ending characters as needed for the target file system (Windows or UNIX).

The problem was that in rewriting the file (using Windows-based StreamReader and StreamWriter functions), the Copy Files task was changing the file encoding to UTF-8. That encoding works fine on Windows and most users didn't even notice. But some users sent me output from file dump tools and comparisons that showed the byte-order mark characters that were added to the file. (SAS users: I knew I could count on you!)

To address this, I changed the "fix line endings" process to use lower level I/O functions that simply scan through the text files as a binary stream, byte-for-byte, and change the line endings as needed. Trying to decide on proper encoding is risky business, so I decided to leave the character encoding untouched.

In addition to my own testing, a couple of users out there have confirmed that my changes fix the issues -- at least for now. Thanks for that! If you want to try the latest, get it now from here:

>> Download the Copy Files task

Related articles

Copying files in SAS Enterprise Guide
Inspecting SAS macro variables in SAS Enterprise Guide

Recent versions of SAS Enterprise Guide (version 5.1 and later) use Microsoft .NET 4.0, which enforces additional security requirements before running custom task DLLs that you download from the Web, including those that you download from support.sas.com. Because these task DLLs are downloaded from the (big and scary) Internet, the Microsoft .NET runtime does not automatically "trust" them as it would trust a properly installed application. To enable the task to run, you must first "unblock" the file using Windows Explorer.

1. Using Windows Explorer, browse to the assembly (DLL) that you downloaded from the samples and extracted from the Zip file.
2. Right-click on the DLL file, and from the shortcut menu, select Properties.
   The Properties dialog box opens.
3. On the General tab, click Unblock to indicate that this DLL is trusted
   (Note: the Unblock button will not appear if the assembly is already unblocked and available.)
4. Click OK to close the Properties dialog box.

If the DLL is blocked when you try to add it in SAS Enterprise Guide, you might see a message such as the following, and the task will not appear in the Tools->Add-Ins menu:

Unable to load program .... 
Could not load file or assembly 'SAS.Tasks.Examples' or 
  one of its dependencies

Related articles

Custom tasks for SAS Enterprise Guide: Q & A
Introduction to SAS Custom Tasks [SAS Talks webinar]
Custom Tasks for SAS Enterprise Guide using Microsoft .NET

Last week I alluded to some very useful applications of the Copy Files task. This is one of them.

Using the SAS programming language, you can manipulate data and create files of just about any size, shape, and format: Excel, PDF, CSV, RTF, and more. A challenge for SAS Enterprise Guide users has been: how to capture those files and bring them back to your local PC, when the SAS Workspace is running on a remote machine?

Example: Export to a CSV file and download the result

Here's a typical scenario: You have a simple SAS program that produces one or more CSV files that you will ultimately use in another program. How can you get the CSV files to your PC automatically?

STEP 1: Build a program step to create the CSV file
This program is easy to adapt for any data set and environment. It works on Windows and UNIX. All you need to know is the library and member name of the data that you want to export, and then the destination folder for your local PC. The program will perform the export operation, stage the CSV file in a temp location, and define the macro variables that the next step will use.

/* Data to export */
%let lib  =         sashelp;
%let datafile =     class;
 
/* Local folder to download to */ 
%let download_to =  c:\projects\data\results;
 
/* detect proper delim for UNIX vs. Windows */
%let delim=%sysfunc(ifc(%eval(&sysscp. = WIN),\,/));
 
%let download_from =
  %sysfunc(getoption(work))&delim.&datafile..csv;
 
filename src "&download_from.";
 
proc export data=&lib..&datafile.
  dbms=csv 
  file=src
  replace;
run;
 
filename src clear;

STEP 2: Use Copy Files task to download the result
The Copy Files task accepts SAS macro expressions. That's a key feature, as the macro variables we need are defined in the previous program step. Here's a screen shot of the task settings:

This makes the use of the Copy Files task very "generic". In fact, you can create a Task Template that defines these exact task settings, and thus always have it available on your Tasks menu directly.

STEP 3: Link these steps together in a process flow
Create a user-defined link between the program and the task, ensuring that they will run in the correct sequence.

THAT'S IT!
The power of SAS and the flexibility of the Copy Files task really makes this a simple operation. However, you might want to consider a few variations:

• Export and download a collection of files in one step. With minor mods to the SAS program, you can loop through a collection of SAS data sets and export multiple CSV files. Instead of defining a single file to download, set the &DOWNLOAD_FROM variable to a file spec with a wildcard. The Copy Files task can handle wildcard notation -- no problem. (Well, no problem anymore, as long as you grab this update.)

  /* specify a wildcard */
  %let download_from =
    %sysfunc(getoption(work))&delim.%str(*).csv;
   
  /* file to create in step */
  filename src "%sysfunc(getoption(work))&delim.&datafile..csv";

• Add a date stamp to your results file. You might have a requirement to keep older versions of your results. With a simple adjustment to the macro expression, you can append a date stamp to the files you create. This will ensure that even if you download the results to the same location each day, the previous results will not be replaced. When you download the file, the name with the date stamp will be intact.

  filename src 
   "%sysfunc(getoption(work))&delim.&datafile._%trim(%sysfunc(today(),date9.)).csv";

  Sample result from this step:
  

  NOTE: The file SRC is:
        Filename=/sas/work/class_19MAY2013.csv,
  

This is just one example of the useful things you can do with the Copy Files task. SAS users are a creative bunch. What other uses can you think of for this task?

Related articles

Copying files in SAS Enterprise Guide
Fixes for the Copy Files task in SAS Enterprise Guide

If you write a blog, you deal with spam comments. That's just part of the deal. Spammers are forever inventing new and creative methods for "tricking" you into accepting their spam comments. These comments have nothing to do with your blog topic but do contain trackback links to their own irrelevant websites. Apparently, adding more links translates to higher ranking on search engines. At least, that's the hope of the spammer.

We've had spammers come to visit ever since SAS blogs began in 2007. As blog authors we moderate each incoming comment, so a spam comment cannot appear on a blog unless an author is duped into approving it. Sometimes spam comments look a lot like legitimate comments, which means that each comment requires some scrutiny.

Evaluating spam comments adds a bit of overhead to a blogger's workload, but until recently this was just a small bit of noise that was mixed in with all of the high-quality blog content and comments that we enjoy with our community of readers. (SAS blogs have accumulated almost 10,000 legitimate comments from engaged readers.)

Then, during November 2012, the spam activity spiked -- at least, that's the way it initially seemed to me. The volume of spam content just felt different to me, but because I was in the habit of marking such comments as "spam" immediately, it was difficult to know for certain how much increased spam volume I was experiencing. Since I'm just one author out of dozens of active blog authors at SAS, I wondered whether others were having the same experience.

Well, I did not have to wonder for very long. We have data: our SAS blogs are hosted on a WordPress platform, which uses a MySQL database to store content and activity. And we have SAS. Using the two together, I produced this:

The red dots indicate the number of spam comments received in a weekly period, and the gray line shows the overall trend. This chart shows clearly that at the end of 2012, the spammers really started going to town.

We know about commercial anti-spam solutions that can be applied to this problem, but we had to ask our IT staff (who supports our blogging platform along with all of our website infrastructure) for a slice of their time to help us.

At SAS, we are lucky to have a tremendously responsive IT team. When we have a business problem that IT can help to solve, they are always happy to partner with us. But they are very busy with lots of projects, all of which compete for resources based on various priorities. As bloggers, we were able to say, "we are receiving a lot more spam than we used to." But what impact does that have, and how should we prioritize the remedy?

I created a report that includes the chart shown earlier plus the one below. You can see the pace at which spam arrives, with sometimes 200 spam messages per day, or over 1000 per week. Even if each spam message requires just a few seconds of attention, the cost (in time) adds up quickly.

I wish I could say that these reports resulted in a "drop everything! solve this!" reaction from IT. Alas, as important as blogs and bloggers are, the SAS IT staff still had bigger fish to fry on other projects. But these reports did provide unambiguous, objective evidence of my hunch: spam was a serious problem and a productivity issue for SAS bloggers. And everyone agreed that something had to be done.

Today, thanks to our IT staff, we have an effective spam solution in place. The spam noise has been reduced down to a slight hum. My historical spam reports are completely broken because spam messages are automatically categorized and deleted. But that's okay -- the reports have served their purpose.

By the way, if you are curious about which blogs seem to attract the most spam, you're reading one right now. The SAS Dummy and The DO Loop both top the list for the most spam accumulated, even though neither of these blogs contains the highest number of articles.

At Rick Wicklin's suggestion I also created a plot of blog-post-volume versus spam-comment-volume. Here's the result, which shows that even though SAS Voices has the highest number of posts, it hasn't attracted nearly as much spam as some of the other blogs. I guess Rick and I each cover topics that the spammers love:

I even used a technique that Rick shared for labeling only certain points within the plot. Wow, these blogs are really useful after all!

One of the great things about SAS libraries is that you can write your programs to read and write data without having to worry about where the data lives. SAS data set on a file system? Oracle table in a database server? Hadoop data in Hive? For many SAS applications, the programmer can treat these very different data sources in exactly the same way.

...except for the cases where you can't treat them the same, because they are different. These different databases have different capabilities and behaviors, and sometimes you need to optimize your SAS programs to take advantage of those differences.

Recently, a colleague at SAS needed a reliable SAS macro-based method to detect the library engine for a given libref. Here's what we came up with:

%macro getEngine(libref);
 %global ENGINE;
 %let dsid=%sysfunc(open(sashelp.vlibnam(where=(libname="&libref.")),i));
 %if (&dsid ^= 0) %then %do;  
   %let engnum=%sysfunc(varnum(&dsid,ENGINE));
   %let rc=%sysfunc(fetch(&dsid));
   %let engine=%sysfunc(getvarc(&dsid,&engnum));
       %put &libref. ENGINE is &engine.;
   %let rc= %sysfunc(close(&dsid.));
 %end;
%mend;

This simple macro peeks inside SASHELP.VLIBNAM, a virtual table that maintains the vital characteristics of all of the active SAS libraries in a session. The macro queries the table for the engine name for the given library, and places the result in a macro variable named &ENGINE.

Here are some example uses and results. The first two calls are for built-in SAS libraries, which use the BASE engine (aliased to "V9"). The third call is for a MySQL library that I use for reporting on our WordPress database.

34         %getEngine(SASHELP);
SASHELP ENGINE is V9
35         %getEngine(WORK);
WORK ENGINE is V9
36         %getEngine(WPBLOGS);
WPBLOGS ENGINE is MYSQL

As part of my follow-up to SAS Global Forum 2013, I've posted a few articles about how to create your own client apps with SAS Integration Technologies. This article shows how to use Microsoft .NET -- the same approach used for SAS Enterprise Guide and SAS Add-In for Microsoft Office -- to assemble a powerful client application.

About Microsoft .NET

Microsoft .NET, when accessed through Microsoft Visual Studio, offers a full application development platform. You can download Express editions of Microsoft Visual C# or Visual Basic .NET for free from the Microsoft website. However, if you want the best productivity with the tool, consider investing in one of the professional editions. If you're just getting started with Microsoft .NET, check out the many resources that Microsoft offers.

How to reference SAS Integration Technologies from Microsoft .NET

To create a Microsoft .NET project that can connect to a SAS environment with SAS Integration Technologies, follow these steps in Microsoft Visual Studio:

1. Create a new project of any type, appropriate for your objective (for example, a Windows Forms application, Console application, or Windows Presentation Foundation application).
2. In your Visual Studio project, add references to the following DLL files:
   • SASOManInterop.dll (for SAS Object Manager classes)
   • SASInterop.dll (for SAS Workspace classes)
   • SASOMIInterop.dll (for SAS Metadata Server classes)
   • SASIOMCommonInterop.dll (supports SASInterop.dll and SASOMIInterop.dll; not used directly in these examples)

   You can find these files in the installation folder for the SAS Integration Technologies client. For example: C:\Program Files\SAS\Shared Files\Integration Technologies or C:\Program Files\SASHome\x86\Integration Technologies.

Here's a screenshot of a Visual Studio project that contains these references:

About the sample application

As a companion to my SAS Global Forum paper, I have prepared a full sample application with these features:

• connects to a SAS Workspace session using a server that you define in a dialog box.
• allows you to connect to a local instance of SAS – no configuration required.
• features three windows: a Program Editor, a log viewer, and a listing viewer. (Does that seem familiar?)
• allows you to run a SAS program on a background thread, keeping the main user interface responsive.
• retrieves the SAS log and listing output, and colors each line of output as appropriate (errors, warnings, notes, page boundaries).
• Provides a simple data table viewer that supports filters.

Here is a screen capture of the application, named SAS Program Harness. It shows a program, the SAS log, and the listing output:

The sample application was built using Microsoft Visual Studio 2012. You can also use Microsoft Visual C# Express (which is free) to view, modify, and build the project. See the GitHub repository to download the project files: source code and ready-to-run binaries.

In future articles I'll focus on some implementation details such as connecting to servers, submitting programs, and viewing data. However, you don't have to wait! You can review the details within the paper right now, and examine the source code on GitHub.

If you've watched any of the demos for SAS Visual Analytics (or even tried it yourself!), you have probably seen this nifty exploration of multiple measures.

It's a way to look at how multiple measures are correlated with one another, using a diagonal heat map chart. The "stronger" the color you see in the matrix, the stronger the correlation.

You might have wondered (as I did): can I build a chart like this in Base SAS? The answer is Yes (of course). It won't match the speed and interactivity of SAS Visual Analytics, but you might still find this to be a useful way to explore your data.

The approach

There are four steps to achieving a similar visualization in the 9.3 version of Base SAS. (Remember that ODS Graphics procedures are part of Base SAS in SAS 9.3, so you don't need SAS/GRAPH for this example!)

1. Use the CORR procedure to create a data set with a correlations matrix. Actually, several SAS procedures can create TYPE=CORR data sets, but I used PROC CORR with Pearson's correlation in my example.
2. Use DATA step to rearrange the CORR data set to prepare it for rendering in a heat map.
3. Define the graph "shell" using the Graph Template Language (GTL) and the HEATMAPPARM statement. You've got a lot of control over the graph appearance when you use GTL.
4. Use the SGRENDER procedure to create the graph by applying the CORR data you prepared in the first two steps.

Here's an example of the result:

The program

I wrapped up the first two steps in a SAS macro. The macro first runs PROC CORR to create the matrix data, then uses DATA step to transform the result for the heat map.

Note: By default, the PROC CORR step will treat all of the numeric variables as measures to correlate. That's not always what you want, especially if your data contains categorical columns that just happen to be numbers. You can use DROP= or KEEP= data set options when using the macro to narrow the set of variables that are analyzed. The examples (near the end of this post) show how that's done.

/* Prepare the correlations coeff matrix: Pearson's r method */
%macro prepCorrData(in=,out=);
  /* Run corr matrix for input data, all numeric vars */
  proc corr data=&in. noprint
    pearson
    outp=work._tmpCorr
    vardef=df
  ;
  run;
 
  /* prep data for heat map */
data &out.;
  keep x y r;
  set work._tmpCorr(where=(_TYPE_="CORR"));
  array v{*} _numeric_;
  x = _NAME_;
  do i = dim(v) to 1 by -1;
    y = vname(v(i));
    r = v(i);
    /* creates a lower triangular matrix */
    if (i<_n_) then
      r=.;
    output;
  end;
run;
 
proc datasets lib=work nolist nowarn;
  delete _tmpcorr;
quit;
%mend;

You have to define the graph "shell" (or template) only once in your program. The template definition can then be reused in as many PROC SGRENDER steps as you want.

This heat map definition uses the fact that correlations are always between -1 and 1. Negative numbers show a negative correlation (ex: cars of higher weight will achieve a lower MPG). It's useful to select a range of colors that make it easier to discern the relationships. In my example, I went for "strong" contrasting colors on the ends with a muted color in the middle.

  /* Create a heat map implementation of a correlation matrix */
ods path work.mystore(update) sashelp.tmplmst(read);
 
proc template;
  define statgraph corrHeatmap;
   dynamic _Title;
    begingraph;
      entrytitle _Title;
      rangeattrmap name='map';
      /* select a series of colors that represent a "diverging"  */
      /* range of values: stronger on the ends, weaker in middle */
      /* Get ideas from http://colorbrewer.org                   */
      range -1 - 1 / rangecolormodel=(cxD8B365 cxF5F5F5 cx5AB4AC);
      endrangeattrmap;
      rangeattrvar var=r attrvar=r attrmap='map';
      layout overlay / 
        xaxisopts=(display=(line ticks tickvalues)) 
        yaxisopts=(display=(line ticks tickvalues));
        heatmapparm x = x y = y colorresponse = r / 
          xbinaxis=false ybinaxis=false
          name = "heatmap" display=all;
        continuouslegend "heatmap" / 
          orient = vertical location = outside title="Pearson Correlation";
      endlayout;
    endgraph;
  end;
run;

You can then use the macro and template together to produce each visualization. Here are some examples:

/* Build the graphs */
ods graphics /height=600 width=800 imagemap;
 
%prepCorrData(in=sashelp.cars,out=cars_r);
proc sgrender data=cars_r template=corrHeatmap;
   dynamic _title="Corr matrix for SASHELP.cars";
run;
 
%prepCorrData(in=sashelp.iris,out=iris_r);
proc sgrender data=iris_r template=corrHeatmap;
   dynamic _title= "Corr matrix for SASHELP.iris";
run;
 
/* example of dropping categorical numerics */
%prepCorrData(
  in=sashelp.pricedata(drop=region date product line),
  out=pricedata_r);
proc sgrender data=pricedata_r template=corrHeatmap;
  dynamic _title="Corr matrix for SASHELP.pricedata";
run;

Download complete program: corrmatrix_gtl.sas for SAS 9.3

Spoiler alert: These steps will only get easier in a future version of SAS 9.4, where similar built-in visualizations are planned for PROC CORR and elsewhere.

Related resources

You can apply a similar "heat-map-style" coloring to ODS tables by creating custom table templates.

If you haven't yet tried SAS Visual Analytics, it's worth a test-drive. Many of the visualizations are inspiring (as this blog post proves).

Finally, while I didn't dissect the GTL heat map definition in detail in this post, you can learn a lot more about GTL from Sanjay Matange and his team at the Graphically Speaking blog.

Acknowledgments

Big thanks to Rick Wicklin, who helped me quite a bit with this example. Rick validated my initial approach, and also provided valuable suggestions to improve the heat map and the statistical meaning of the example. He pointed me to http://colorbrewer.org, which provides examples of useful color ranges that you can apply in maps -- colors that are easy to read and don't distract from the meaning.

Rick told me that he is working on some related work coming up on his blog and within SAS 9.4, so you should watch his blog for additional insights.

Many SAS Enterprise Guide users practically live in the Query Builder. For those who understand their data tables, the Query Builder provides a tremendous amount of flexibility to pull and manipulate data. The Query Builder produces SQL programs behind the scenes, which translates well for database-centric work.

Sometimes a complex query requires a two-pass process. For example, suppose that you want to create a table of CARS that contains only those models with a higher-than-average price. First you need a query to calculate the average price across all cars. Then you need a second query to filter the records with a price that's higher than that result.

A SAS Enterprise Guide user might build something like this:

This method has obvious downsides: it requires two passes through the data, and two sets of results are brought into your SAS session. If the source table is in a database, this can be especially inefficient.

SAS Enterprise Guide 5.1 supports the idea of subqueries, a concept familiar to SQL programmers. With a subquery (sometimes called "subselect" or "inner query"), you can express the entire operation in one SQL step. That's one trip to the database, and only one set of results brought back to SAS.

I'll briefly describe how this works. For comprehensive documentation and examples about the subquery feature in SAS Enterprise Guide 5.1 and later, read Michael Burke's excellent conference paper on Finding Your Inner Query with SAS Enterprise Guide.

The subquery feature relies on the ability to define query templates (also new in SAS Enterprise Guide 5.1), which are query definitions that you can reuse as building blocks for other query steps. (Query templates are useful on their own, as a way to store a query within your task menus so that you can use it across different projects.)

Using the subquery feature

First, use the Query Builder to define the "inner query". In my example, that query contains just one item, the calculated average of MSRP.

Instead of running the query, use the drop-down arrow on the Run button to select Create Template. The Create Template window allows you to assign a name to this definition, and it's automatically added to your Tasks->Task Templates menu. Note that you can create the query template without even running this part of the query. That can be a time saver if you're a Query Builder Ninja with high confidence in your result.

Next, design the outer query with a new Query Builder task. Select all of the fields that you want to include in the result. In the Filter tab, add a new filter for MSRP. Instead of comparing MSRP to a constant or other expression, you'll select from a list of available Subquery templates.

In your Filter tab, you'll see a partial view of the inner SELECT expression.

When you complete your selections and have the other query options just the way that you want them, click Run to execute the query.

If you examine the SAS program that the task creates, you'll see the inner SELECT clause:

PROC SQL;
   CREATE TABLE WORK.expensive AS 
   SELECT t1.Make, 
          t1.Model, 
          t1.MSRP, 
      FROM SASHELP.CARS t1
      WHERE t1.MSRP >= (
         SELECT /* AVG_of_MSRP */
           (AVG(t1.MSRP)) AS AVG_of_MSRP
             FROM SASHELP.CARS t1
      );
QUIT;

You can also use subqueries as a way to filter/compare a column to a list of values. In that case, your inner query simply needs to have a single-column list of results (instead of a single number, as in my example). Read Michael's subquery paper for all of the possibilities and limitations. Here's an executive summary from Michael's paper:

Here are the types of subqueries that you can create using query-based templates:

• Subqueries that return a single value
• Subqueries that return multiple values (multiple rows of a single field)
• Subqueries that appear as part of a filter of the raw data (on the WHERE clause)
• Subqueries that appear as part of a filter on the grouped data (on the HAVING clause)
• Subqueries that appear as part of a recode condition as part of recoding a column in a computed column (on the SELECT clause)

Here are the types of subqueries that you cannot create using query-based templates:

• Subqueries that form a derived table (subqueries that appear on the FROM clause)
• Subqueries that refer to columns on the outer query (correlated subqueries)

More than one way to skin this data

There are many other ways to accomplish this simple example in SAS. For example, you may be able to use the HAVING clause feature in the Query Builder, or use SQL joins and special join conditions to filter a table based on another result set. But the ability to create a subquery has been long-requested from SAS Enterprise Guide users who understand databases. I hope you'll agree that it's an important addition to your query and reporting toolbox.

When I work on SAS projects that create lots of files as results, it's often a requirement that those files be organized in a certain folder structure. The exact structure depends on the project, but here's an example:

/results
   |__ html
       |__ images
   |__ xls
   |__ data

• Simply require that the folders exist before you run through the project. (This is the SEP method: Somebody Else's Problem.)
• Use SAS statements and shell commands (via SYSTASK or other method) to create the folders as needed. The SAS-related archives are full of examples of this. It can get complex when you have to account for operating system differences, and whether operating system commands are even permitted (NOXCMD system option).

In SAS 9.3 there is a new system option that simplifies this: DLCREATEDIR. When this option is in effect, a LIBNAME statement that points to a non-existent folder will take matters into its own hands and create that folder.

Here's a simple example, along with the log messages:

options dlcreatedir;
libname newdir "/u/sascrh/brand_new_folder";

NOTE: Library NEWDIR was created.
NOTE: Libref NEWDIR was successfully assigned as follows: 
      Engine:        V9 
      Physical Name: /u/sascrh/brand_new_folder

In order to create a series of nested folders, you'll have to create each folder level in top-down order. For example, if you need a "results" and a "results/images" folder, you can do this:

%let outdir=%sysfunc(getoption(work));
/* create a results folder in the WORK area, with images subfolder */
options dlcreatedir;
libname res "&outdir./results";
libname img "&outdir./results/images";
/* clear the librefs - don't need them */
libname res clear;
libname img clear;

Or (and this is a neat trick) you can use a single concatenated LIBNAME statement to do the job:

libname res ("&outdir./results", "&outdir./results/images");
libname res clear;

NOTE: Libref RES was successfully assigned as follows: 
      Levels:           2
      Engine(1):        V9 
      Physical Name(1): /saswork/SAS_workC1960000554D_gsf0/results
      Engine(2):        V9 
      Physical Name(2): /saswork/SAS_workC1960000554D_gsf0/results/images

If you feel that folder creation is best left to the card-carrying professionals, don't worry! It is possible for a SAS admin to restrict use of the DLCREATEDIR option. That means that an admin can set the option (perhaps to NODLCREATEDIR to prohibit willy-nilly folder-making) and prevent end users from changing it. Just let them try, and they'll see:

13         options dlcreatedir;
                   ___________
                   36
WARNING 36-12: SAS option DLCREATEDIR is restricted by your Site 
Administrator and cannot be updated.

Exact documentation for how to establish Restricted Options can be a challenge to find. You'll find it within the Configuration Guides for each platform in the Install Center. Here are quick links for SAS 9.3: Windows x64, Windows x86, and UNIX.

Inspired by the JMP blog - Statisticians: harbingers of doom?:

Enjoy what's left of the International Year of Statistics -- while you still can.